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“Research on issues of social justice could not be more timely, urgent, and fundamental. This 
volume provides clear and thoughtful guidance on how we can use and learn from social 
science research to utilize the power of numbers for social good. This volume takes the reader 
through every stage of the research process, including design, data collection, data analysis, 
and community-based scholarship. Undoubtedly this book will be foundational for scholars 
and students who want to use their research for social justice.” 
—Mary Beth Oliver, Bellisario Professor of Media Studies, 
Penn State University, and President Elect, International Communication Association 


“This isn’t your father’s methods text, and that’s a good thing. If you’re focused on social 
justice and suspicious of quantitative methods, this book will show you how quantitative, 
mixed, and qualitative methods can—and should—all be marshalled to address the important 
questions we face today. This is an introduction to quantitative research design and execution 
that successfully engages with societal problems without sacrificing any of the nuance and 
precision expected from a first-rate textbook.” 

—Larry Gross, Professor, University of Southern California 


“This is not just another stats book! Scholarly yet accessible and practical, Scharrer and 
Ramasubramanian critique the misuse of numbers and, more importantly, show us how to 
harness the power of numbers for social justice. This is a book that many in our field have 
been waiting for.” 

—Sonia Livingstone, London School of Economics and Political Science 


“In covering the nuts and bolts of quantitative research methods while showing why and how 
research matters for society, this book is a tremendously valuable resource for students and 

scholars in Communication and across the social sciences.” 
—Dana Mastro, Associate Dean of the College of Creative Studies and Professor of 
Communication, University of California, Santa Barbara 
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This textbook is an advanced introduction to quantitative methods for students in 
communication and allied social science disciplines that focuses on why and how to conduct 
research that contributes to social justice. 

Today’s researchers are inspired by the potential for scholarship to make a difference for 
society, to push toward more just and equitable ends, and to engage in dialogue with members 
of the public so that they can make decisions about how to navigate the social, cultural, and 
political world equipped with accurate, fair, and up-to-date knowledge. This book illustrates 
the mechanics and the meaning behind quantitative research methods by illustrating each step 
in the research design process with research addressing questions of social justice. It provides 
practical guidance for researchers who wish to engage in the transformation of structures, 
practices, and understandings in society through community and civic engagement and policy 
formation. It contains step-by-step guidance in quantitative methods—from conceptualization 
through all the stages of execution of a study, including providing a detailed guide for statistical 
analysis—and demonstrates how researchers can engage with social justice issues in systematic, 
rigorous, ethical, and meaningful ways. 

This text serves as a core or supplementary textbook for graduate and advanced 
undergraduate courses in research methods for communication and social sciences and fills a 
gap for a methods text that is responsive to the desire of scholars to conduct socially impactful 
research. 
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Preface 


We write this preface in September 2020, one day after U.S. Supreme Court Justice Ruth 
Bader Ginsburg has died. We write in the wake of egregious incidents of police brutality 
against Black people and a resulting renewed groundswell of protest and action to honor Black 
lives. We write as wildfires fueled by climate change rage in California, months after more 
than 12 million acres were similarly destroyed in Australia. We write in the midst of a global 
pandemic that has laid bare enormous economic, gender, and racial inequalities. We write in 
a time when the world’s struggles are both hard to bear and impossible to ignore. 

The idea for this book came years ago, however, when it became clear to us that today’s 
social science researchers are often inspired not just by knowledge for its own sake but by the 
potential for research to help make a difference in the world. We have observed emerging 
and established researchers motivated to focus their efforts toward advancing a more just and 
equitable society. We have seen increasing numbers of scholars eager to engage in dialogue 
with members of the public and in advocacy and action toward social change. And although 
we write at a particularly poignant moment in history, of course, every moment in history 
has its own conditions in which to examine issues of (in)justice and (in)equality in society. 

Under its auspices of “objectivity” and the scientific method, at first glance quantitative 
research may seem an unlikely set of tools to use toward social justice ends. Indeed, in Com- 
munication as in other fields, there is a much longer and stronger history of employing quali- 
tative methods in social justice research. Quantitative research, at its heart, is about numbers, 
and in this book we show that numbers have an important role to play when social, eco- 
nomic, and political issues are at stake. We show how the numbers generated from quantita- 
tive research can be used to bring social issues and injustices to light as well as to investigate 
how to address them. We believe quantitative research has an important role to play, and this 
book is designed to provide you, the reader, with the tools needed to engage as a quantita- 
tive researcher with social justice issues in systematic, rigorous, ethical, and meaningful ways. 

Despite the potential power of numbers, however, there is, of course, no magic bullet 
solution to the long-standing and complex issues and injustices that mark our society. And, 
indeed, just as we argue there is particular strength of numbers, there are also particular weak- 
nesses of numbers and of quantitative methods in general to consider from a social justice point 
of view. We acknowledge and explore these limitations alongside our discussion of strengths, 
providing what we hope will be seen as an honest and clear-eyed account of the possibilities 
as well as the barriers involved in designing and using quantitative research for social justice. 

In this book, we introduce quantitative methodologies to show the reader not just how 
to conduct quantitative research but also why to do so. To show how, we give step-by-step 
guidance in the use of quantitative methods, from conceptualization through all stages in the 
execution of a study. In fact, we designed this book to be a comprehensive introduction to 
quantitative methods that provides a level of detail (especially, but not exclusively, regarding 
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the use of statistical analysis) not always found in an introductory methods book. When we 
show how to do quantitative research, we do so from a social justice perspective, exploring 
how to design and carry out each step with care and concern for issues of access, diversity, 
equity, and inclusion. To show why to conduct quantitative research, we give examples of 
quantitative research positioned as a means of challenging the structures, policies, and prac- 
tices that stand in the way of greater justice and equality. We illustrate ways to ensure research 
does not remain solely in the academy but can also be put to good use by citizens and change- 
makers alike. We use real-world case studies and research materials, self-reflexive interviews 
with scholar-activists, and an easy-to-follow writing style to introduce quantitative social 
scientific research methods for social justice. Overall, then, this book illustrates the mechanics 
as well as the meaning behind quantitative research conducted to be responsive to and even 
to intervene in social injustices and inequalities and to promote inclusion, equity, and justice. 

We wrote this book from our position as Communication researchers, and, therefore, the 
examples that we highlight in this book are from the communication and media studies field. 
However, the principles and practices that we introduce apply well beyond Communication 
to other social sciences. We also write from our position as active researchers ourselves, hav- 
ing accrued a number of experiences relevant to the kinds of research/public engagement/ 
activism that we introduce in this book. 

The writing of this book was a true collaboration between the two authors. We split the 
work equally and we each poured over every sentence and every figure you will see. We sup- 
ported each other and found a way forward, together, after our original publisher dropped 
our project when the writing was nearly done, despite a signed contract. We are thrilled to 
have found a home with Routledge and pleased with what we produced together. Part of 
our pleasure in writing this book stems from our observation that it is rare that a quantitative 
methods book is written by women scholars, and even more rare for one of those scholars to 
be a person of color. 

Here is a chapter-by-chapter account of what you will find inside: 

Chapter 1 focuses on defining key concepts and terms such as social justice, activism, and com- 
munity engagement, and introduces the defining features of quantitative research. We address 
why Communication is a good position from which to conduct research for social justice, 
and we begin to explore key principles in the possibilities and limits of quantitative research 
in Communication for social justice. What are the assumptions in quantitative research about 
ways of knowing, whether the researcher is trying to be objective, whether the social world 
can be measured, etc.? How do the foundations of and assumptions behind quantitative 
research allow the researcher to work toward social justice? What constraints might those 
foundations and assumptions introduce? 

In Chapter 2, we examine the ethical considerations that quantitative Communication 
researchers must face. What ethical considerations arise, generally, in conducting quantita- 
tive Communication research? What particular ethical considerations stem from quantitative 
Communication research conducted with a goal of contributing to social justice? How can 
researchers navigate these ethical questions and challenges? In this chapter, we introduce 
human subjects protections—ensuring the well-being of the people who are participating in 
research—and other ethical considerations involved in quantitative Communication research 
for social justice. 

In Chapter 3, we move on to the first of many of the nitty gritty details of designing and 
implementing quantitative research projects addressed in the book. Chapter 3 deals with 
the practice of “operationalization.” How does the researcher move from abstract definitions 
of concepts that they want to study to something that can be measured using quantitative 
methods? How can we operationalize concepts (which is what we call it when we come up 
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with ways to measure those concepts) and identify variables (which are the specific features 
of something that we are studying) in a way that recognizes the range of human differences 
and the complexities of human experiences? How do we measure variables in a way that is 
reliable, meaning it is likely to produce stable and consistent results? How do we measure 
variables in a way that is valid, meaning it is doing a good job of tapping into the meaning of 
the concept? 

Sampling is the focus of Chapter 4. In research, it is quite rare to study the entire popula- 
tion of interest. Instead, we usually sample from that population to make the size of who or 
what we are studying a bit more manageable. What are the principles and techniques behind 
sampling? How can we decide which units will be examined and which will not in a way that 
is responsive to the principle of inclusion as well as to the scientific method? How can we 
determine who or what will be present or absent in our sample or in our research in general? 
In this chapter, we consider the interplay between the principles of sampling and the princi- 
ples of social justice. 

In Chapter 5, we take up one quite popular quantitative method, the survey, to discuss in 
detail. The chapter answers a number of questions, including: What are the steps involved in 
designing and carrying out a survey study? Why might you want to use a survey design in 
the first place? What are examples of survey studies in Communication that advance social 
justice? How can you design a survey in a way that attends to the social justice principles of 
diversity and inclusion? What are the weaknesses of survey research for researchers who want 
to conduct research for social justice? 

Experiments are the topic of Chapter 6, defined as tools for quantitative social scientists to 
use to explore how people respond to a stimulus—something that they see, or an event they 
are asked to participate in—in the short term. This chapter provides a step-by-step guide in 
how to conduct an experiment and addresses why one might want to choose this method. 
We also discuss advantages and disadvantages of this particular method from a social justice 
perspective. 

Similarly, Chapter 7 examines content analysis methodology in depth. Content analysis is 
a research technique in which the features of texts are studied in a systematic manner, and 
this method, too, is quite common in Communication research. In this chapter, we take the 
reader through a step-by-step exploration of how to design and carry out a content analysis 
study, and we share examples of existing studies that illustrate representations of social groups 
in the media and other important social justice-related topics. 

Given that quantitative research data take the form of numbers, a key step in research meth- 
ods is to use statistical analysis to make sense of those numbers, imbuing them with meaning 
and interpretation. Chapter 8 describes the principles associated with analyzing quantitative 
data, answering questions such as: How can researchers make an inference from data observed 
in a sample to the larger population? How do researchers know whether their hypotheses are 
supported? What claims can be made from statistical results, and what caveats must be kept 
in place? 

Chapter 9 extends the principles discussed in Chapter 8 into the practical steps involved in 
using statistical software to analyze quantitative data. Introducing readers to SPSS (Statistical 
Package for Social Sciences), complete with a practice data set to use and a series of screen- 
shots to illustrate, Chapter 9 provides a detailed guide on how to run several common statisti- 
cal tests and how to make sense of the results that are produced from those tests. 

Chapter 10 introduces the researcher to the possibilities that exist for combining quantita- 
tive and qualitative research, known as mixed methods research. How does the researcher 
design a study that combines a method from the quantitative toolkit with one that comes 
from the qualitative toolkit? Why might one want to utilize such an approach? We review 
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examples of mixed methods studies in Communication research for social justice in this chap- 
ter and discuss the advantages as well as the challenges associated with such methodological 
triangulation. 

The final section of this book moves from research design and data analysis to ways to 
partner and collaborate with various publics such as community members, practitioners, and 
policymakers for greater impact and action. In Chapter 11, we discuss community partnerships 
and collaboration. How can we form collaborations with community organizations or other 
partners in our research that will last and that will satisfy each other’s priorities and needs? 
What are the benefits and risks of doing so? What makes such partnerships strong, impactful, 
and mutually beneficial? This chapter discusses these sorts of collaborations that are often at 
the heart of Communication research for social justice activism. 

Chapter 12 is about the ways that quantitative research can be written to pursue publication, 
to communicate with the public, and to contribute to policy making. What does a success- 
ful quantitative research paper look like? What can one expect from the publication process? 
How can researchers take steps to ensure that they are communicating research not just for 
fellow researchers but for the public? What outlets are available for engaging in dialogue 
about research findings that bear on issues of social, political, and cultural significance of the 
moment? How is quantitative Communication research used to shape policies that impact 
questions of social justice? The final chapter provides tools for researchers on each of these 
important forms and purposes for communicating research. The ability of research to make 
a meaningful impact depends a great deal on who can access and act on that research, and 
therefore this final chapter is a logical conclusion to the steps involved in carrying out research 
and the principles behind what such research can achieve. 

In summary, we have created what we hope you will find to be a highly readable and 
engaging text that is both practical and meaningful. We believe that by framing each and every 
discussion in this introduction of the principles and practices of quantitative research design 
around issues and topics that really matter, we offer a uniquely valuable learning tool. Upon 
reading this book, we hope that readers will be both inspired and sufficiently informed to 
harness the “power of numbers” inherent in quantitative research to illuminate and intervene 
in the most pressing social issues of the day. 
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All around us, there are serious social issues facing our global community. Injustices and ine- 
qualities span a whole spectrum of topics from everyday sexism, food insecurity, homelessness, 
and forced migration to bonded labor. For instance, about 40.3 million people are owned 
by an employer who treats them as property to be sold or bought through human traffick- 
ing (Walk Free Foundation, 2018). Over 79.5 million people have been forced to leave their 
homes due to conflict and persecution, and over 40% of these people are children (United 
Nations Refugee Agency, 2020). One in nine people around the world do not have enough 
food to lead an active and healthy life (Food Aid Foundation, 2019). In the United States, 
working women make only 81 cents to the dollar compared to men (United States Census 
Bureau, 2020), and systemic and institutionalized racism continues to produce conditions in 
which prisons are overpopulated with men of color (The Sentencing Project, 2019). Only 
23.7% of U.S. congressional seats are held by women, and women make up just 29% of hold- 
ers of statewide elected executive offices (Center for American Women and Politics, 2020). 

Each of these startling figures that we have shared is brought to light due to quantitative 
research, a way to produce knowledge that uses scientific methods to observe particular phe- 
nomena. Yet, alongside the distressing numerical estimates described here, community mem- 
bers have also used their assets and resources to resist, challenge, and resolve social injustices 
and inequalities in powerful ways. The high school student survivors of the Parkland school 
shootings in Florida have used their voices powerfully and raised over three million dollars 
to challenge the role of the National Rifle Association in U.S. politics. The Women’s March 
in 2017, as the largest single-day protest in recorded history, was a worldwide movement to 
bring attention to human rights and women’s health, education, racial justice, LGBTQ+t+ 
rights, employment, wages, and other gender-based discrimination. LGBTQ+ activists have 
fought inside and outside courtrooms to battle homophobia, and same-sex marriages are now 
legalized in many countries around the world. The Black Lives Matter movement has suc- 
ceeded in challenging the discourse surrounding police violence against people of color and 
has brought the language of systemic racism and anti-Black racism into the mainstream. Many 
institutions and organizations took steps toward anti-racism commitments and actions in the 
aftermath of the gruesome murder of George Floyd in the U.S. in 2020, an event that led to 
a global uprising. Quantitative research has documented each of these phenomena, as well. 

We raise these opening points to make three observations. First, statistics can be powerful 
tools to use to bring sobering inequalities and injustices to light. Second, statistics and the 
quantitative research that produces them can mark and measure progress made toward equality 
and justice, as well. Third, however, as powerful as numbers such as these may be to document 
injustice and inequality and study steps toward change, statistics can also be considered only a 
partial truth. There is always more to the story than statistics can convey. 
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In quantitative research, numbers have a key role to play, as statistics are used to describe 
and explain what researchers have observed. Numbers can indeed be powerful, since they can 
be used to convey a broad set of truths experienced by many, as you can see in the figures 
and estimates we have shared. Numbers can shape public opinion, organizational decision- 
making, community priorities, and public policies. Of course, numbers can be fabricated, 
decontextualized, and misinterpreted, too. Therefore, having the ability to interpret, discern, 
design, and apply quantitative research is an important skill to learn as Communication schol- 
ars and practitioners. 

Enter the researcher into the scene. As scholars and researchers, what is our role in address- 
ing issues of social justice? How can we make this world a better place to live in through our 
scholarship and research? We believe it is important for researchers to use scientific knowledge 
to address important social problems at the local, national, and global levels. Developing a 
social justice mindset, applying it to research design, and using research results to intervene in 
forces and factors that create and maintain injustices is vital. Indeed, stories shared about anti- 
Black racism via #CommunicationSo White and also the #Blackinthelvory campaign started 
by two Communication scholars, Shardé M. Davis and Joy Melody Woods, reveal the racial 
injustices and White supremacy, among many other inequalities, that need to be dismantled 
even within academia. Rather than being out of touch with the realities of people’s lives, the 
“power of numbers” approach ensures that researchers and the knowledge they produce are a 
resource for the betterment of marginalized groups, especially when stakes are high, whether 
inside or outside of university settings. 


Purpose of Social Science Research 


Research methods are techniques and approaches used to produce knowledge. Knowledge, 
for its part, can be created in a number of ways, each with its own underlying assumptions, 
values, and internal logic. There are many types of research, but this book will focus on social 
science research methods. Social science research methods focus on producing knowl- 
edge that helps us understand the social world around us. 

Whereas basic research is mainly motivated by researchers’ curiosity and interest in pro- 
ducing and theorizing knowledge, applied research seeks to solve real-world practical prob- 
lems. Basic research often assumes a traditional scientific model where the focus is on scientific 
discovery and curiosity. Applied research focuses on how new knowledge is connected to the 
outside world in ways that are useful to the community. However, the relationship between 
basic and applied research is often quite close, with each side potentially informing the other. 
Not all applied research, though, can be called social justice research, unless the research is 
explicitly applied toward identifying and/or reducing social inequalities. As you will see later 
in this chapter, social justice research is focused expressly on challenging imbalances in power, 
position, and privilege and the conditions that produce or maintain those imbalances. 

When it comes to doing research for social justice, simply creating knowledge is not always 
sufficient. What do we do with what we know? Researchers working toward social justice also 
want research to inspire, facilitate, or inform action that could bring about positive changes. 
They can do so using research that is exploratory, descriptive, evaluatory, explanatory, or 
transformative in nature. Sometimes within the same project, there can be multiple goals and 
purposes at different stages of the research process. 


¢ Exploration: Social research can help explore new or fairly under-studied phenomena 
and social issues. Reviews of research findings and personal insights can reveal existing 
gaps in social justice research that need to be filled. 
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Figure 1.1 The idea expressed in this cartoon—that academics only study the world rather than try to save 
it—is an increasingly outdated notion 


Source: Frits Ahlefeldt-Laurvig 
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e Description: Social research can help provide detailed descriptions and systematic 
observations relating to social events, practices, groups, situations, beliefs, and behaviors. 
For instance, content analysis, a method you will read about in Chapter 7 of this book, 
can help to describe patterns within media content that exclude or misrepresent social 
groups. 

¢ Evaluation: Social research can help assess the effectiveness of a program or initiative 
or policy. Within social justice research, especially in community-based interventions or 
organizational communication, this is an important purpose for doing social scientific 
research. 

¢ Explanation: Social research can help explain why social phenomena occur. For instance, 
research methods like surveys and experiments—which you will also read about in Chap- 
ters 5 and 6—help determine whether one point of view, experience, or phenomenon is 
related to another. If it is, we can say that knowing one helps explain the other. 

¢ ‘Transformation: Social research can be used for transformation and positive change 
through social action and activism to impact communities, organizations, and policies. 
Chapter 12 will give more detail about this. 


Approaches to Social Science Research 


There are many approaches to social research: (1) quantitative research, (2) qualitative 
research, (3) mixed methods research, and (4) community-based research. The two main 
types of approaches to social science research methods are quantitative and qualitative research 
(Figure 1.2). Mixed methods research combines qualitative and quantitative research. 
Community-based research involves the community as the primary setting and focus of 
research. There is considerable overlap among these various types of social research methods. 
Also, these are broad umbrella terms that encompass many specific types of methods. 


Quantitative Research 


Quantitative research, the subject of this book, offers one set of tools and techniques to gather 
data and contribute to what we know as well as what we do with what we know. It focuses on 
measuring variables, testing hypotheses, and explaining relationships. This approach empha- 
sizes numerical data, categorization of data, the principle of objectivity, deductive reasoning, 
representativeness, and generalizability of findings, all of which are explained further in this 
chapter and/or throughout this book. Some examples of quantitative research methods are 
content analysis, lab-based experiments, field experiments, and survey methods, and each of 


Quantitative Qualitative 
Measuring variables, testing hypothesis, and Understanding experiences, discovering 
Focus a i i : ah 
explaining relationships meaning, and building theory 
Vaea Objectivity, deductive reasoning, Subjectivity, inductive reasoning, theory 
representativeness, and generalizability building, complexity and depth 
Examples Content analyses, lab-based experiments, and Interviews, focus groups, ethnography, and 


survey methods textual analyses 


Figure 1.2 Quantitative vs. qualitative research 


Source: Emily Riewestahl 
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these methods will be discussed in detail in this book as well. In the process of carrying out 
research with an eye toward social justice, quantitative methods pose both real challenges as 
well as quite fruitful opportunities. Quantitative methods have a number of distinguishing 
characteristics that set them apart from other ways of knowing, as you will see later in this 
chapter. 


Qualitative Research 


Qualitative research focuses on subjective experiences and meaning-making to understand 
and describe social phenomena. It generally uses an inductive approach to research where 
the ideas about what will be discovered or observed as well as the design are flexible and not 
fixed ahead of time (with “ahead of time” often being noted as a priori). The emphasis of 
qualitative research is on in-depth knowledge, “thick” or rich descriptions, and sense-making 
of the complex social world around us. This research approach includes methods such as 
textual analysis, ethnography, focus group discussions, in-depth interviews, case studies, and 
arts-based research methods. More details about qualitative research methods are covered in 
Chapter 10, the chapter on mixed methods. 


Mixed Methods Research 


Mixed methods research combines quantitative and qualitative methods into a single project. 
Quantitative and qualitative methods could be used sequentially at different stages of the 
project or can be integrated simultaneously at the same stage of the project. Mixed methods 
approaches are commonly used in social justice research to explore, understand, evaluate, and 


Quantitative Methods Qualitative Methods 


What did you feel 
wheh you saw the 
free ice cream! 


Only one in 30 takes 
the free ice cream. 
Interesting... 


Excited, 
A little scared, 


And why was that! 


Figure 1.3 Quantitative research identifies numerical patterns across many cases 


Source: Matt Lavoie 
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bring about collective action and social change. We will return to mixed methods research in 
Chapter 10. 


Community-Based Research 


Community-based research uses the community as the place or setting for knowledge creation 
rather than in a university, corporation, or government agency. It can incorporate quantita- 
tive, qualitative, or mixed methods approaches. Community-based research typically involves 
researchers forming community partnerships and collaborations. The community is often 
actively involved in the research design and implementation to address local social issues to 
improve some aspects of the community such as health care, conflict resolution, or education. 
It involves collaborative partnerships among community members, practitioners, and academ- 
ics. The existing strengths and resources in the community are leveraged by the researcher and 
community members to work together to address social issues. For instance, an organizational 
communication scholar might organize public forums for community members on topics 
such as food insecurities or disputes over shared resources such as water. We devote Chap- 
ter 11 in this book to community-based research. 


Research Paradigms 


Essential to understanding the role of quantitative research in social justice is an explora- 
tion of the central assumptions and practices that underlie quantitative inquiry. There are 
many approaches to research and multiple “ways of knowing.” Epistemology is a term 
that encompasses how knowledge is conceived and produced. To better understand the 
epistemology of quantitative research for social justice, some questions that we can raise are: 
How do we produce what will come to be known as knowledge? What steps will research- 
ers take to produce knowledge in a manner that is convincing and defensible? How do we 
justify a claim or an observation as constituting knowledge? On what grounds does a claim 
of knowledge rest? What assumptions underlie such claims? Who are seen as knowledge 
producers and why? 

These questions bring us to a discussion of paradigms and the way that they help structure 
research and the creation of knowledge. A paradigm is a “set of interrelated assumptions 
about the social world which provides a philosophical and conceptual framework for the 
organized study of that world” (Filstead, 1979, p. 34). In other words, paradigms are ways to 
map out different approaches by thinking about what they have in common and how they 
differ from one another. Communication research has largely stemmed from three paradigms: 
(1) positivist, (2) constructivist, and (3) critical paradigms. Each is described in detail in the 
following text. 


Positivist and Post-Positivist Paradigm 


Quantitative research has been most closely connected to positivism (Wrench, Thomas-Mad- 
dox, Richmond & McCroskey, 2008). Positivism is adopted from the natural sciences and 
involves generating predictions in the form of hypotheses and then gathering data to test those 
hypotheses. This is, by definition, a deductive process as the researcher begins with a general- 
ized principle and then observes specific data to test that principle. It stands in contrast to an 
inductive approach that begins with specific data that will come to form a generalized principle 
or set of principles. Positivism traditionally imagines a single, measurable reality that sound 
research design can reveal, and it detaches the researcher from what and who is studied. In 
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other words, under assumptions of positivism, different researchers will come to just about the 
same conclusions if they use similar approaches to observe the world around them. 

Important for the central premise of this book, for some, positivism has evolved to encom- 
pass a slightly different set of approaches, assumptions, and understandings better defined 
as post-positivism. Post-positivism retains the uses of hypotheses and deductive reasoning 
characteristic of the scientific method but abandons the notion of a single reality to be dis- 
covered and acknowledges that researcher subjectivity and influence are unavoidable (Pop- 
per, 1968). Although post-positivists accept that their own values and beliefs are influencing 
the questions they ask, where and how they ask them, and how they are interpreting results, 
rather than fully embracing a subjective position in knowledge creation they seek to minimize 
the biases that those subjectivities introduce. 


Constructivist Paradigm 


In contrast, constructivism considers multiple realities and meanings built from varied expe- 
riences and interpretations rather than a single truth that is “out there” for a researcher to 
discover. The researcher is much more closely connected to the research process as a subjec- 
tive interpreter or even participant rather than as a detached observer striving to be objective. 
Researchers may not arrive at similar observations, then, since their own subjective interpre- 
tations are thought to shape how they interpret what they see. In qualitative research words, 
stories, texts, and images comprise the observations made, whereas quantitative research uses 
numbers to represent data. Qualitative approaches rather than quantitative approaches are 
typically a better fit for those operating within the constructivist paradigm because words, 
stories, texts, and images, rather than numbers, can often better reflect the potentially limitless 
individual meanings, experiences, and points of view of research participants. 


Critical Paradigm 


In the critical paradigm, the critique of the distribution of power and privilege in society is at 
the center of the inquiry, and research is positioned as a means of direct intervention in oppres- 
sion, injustice, and inequality. In other words, “an emancipatory principle drives such research” 
(Denzin, 1994, p. 509). In Communication as well as in other disciplines and fields, critical 
research has been much more closely associated with qualitative rather than quantitative meth- 
ods. That is because, in the critical paradigm, the researcher has a direct interest in disrupting 
the status quo of oppression rather than being rather detached, personally, from what is being 
observed. It is partly for this reason that some find critical research to be at odds with the logic, 
assumptions, and techniques of quantitative methods (Guba & Lincoln, 1994), a complex issue 
that we will take up in great detail both in this chapter and in the rest of this book. 

These three paradigms (or slight variations of them) have been used to organize the intel- 
lectual space that Communication research occupies for quite some time. Yet, these paradigms 
have also shifted a bit as well. Paradigm shifting has been discussed extensively in regard to 
scientific knowledge (Kuhn, 1962). Such shifting of the rules and assumptions that underlie 
knowledge creation and the organization of knowledge makes space for the possibility of 
social justice research using quantitative methods, the subject of this book. 


Types of Social Justice Research 


Social justice research has the primary goal of challenging oppression, addressing social 
inequalities, and working to correct power imbalances (Carragee & Frey, 2016; Frey & 
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Carragee, 2007). It focuses on marginalized groups who are “economically, socially, politi- 
cally, and/or culturally under-resourced” (Frey, Pearce, Pollock, & Murphy, 1996, p. 110). 
Social justice research is often interdisciplinary, multi-method, and multi-perspectival. 
The overarching theme amongst social justice researchers is a desire to “make a differ- 
ence” (Frey, 2009). While there are many ways to make a difference through research, the 
important questions to ask are “Who does this research serve?” and “Why do we engage 
in this research?” 

By nature, social justice research uses the critical paradigm in its approach. It questions 
power differences (Who has power and privilege? Who does not?) and works toward reducing 
inequalities. It challenges systems of domination such as patriarchy, heteronormativity, racism, 
imperialism, capitalism, and ableism that often intersect to form multiple layers of oppres- 
sion shaping individuals’ lived realities (Collins, 2000; Collins & Bilge, 2016; hooks, 2000). 
Social justice research could, for instance, highlight poor working conditions, unequal pay 
across genders, negative stereotypical portrayals of racial/ethnic minorities, sexual harassment 
in organizations, ableism in educational institutions, online anti-LGBTQ rhetoric, capitalistic 
motives in climate change policies, or colonial antidemocratic media flows. 

There are many types of social justice research, with slight variations but some overlap 
among types. Translational research aims to make academic research more accessible 
and relevant to a non-academic lay audience (Tracy, 2002). Public scholarship is similar 
to translational research in its goal of educating the public about academic scholarship. 
However, it also seeks to listen to public voices and incorporate their feedback into scholar- 
ship. Engaged scholarship bridges the gap between theory and practice by making aca- 
demic research more relevant to practitioners and communities. Knowledge is co-produced 
through collaborative processes involving academics and a variety of practitioners (Barge & 
Shockley-Zalabak, 2008). Policy research applies academic expertise to analyze, evalu- 
ate, and formulate policies addressing specific issues. Community-based participatory 
research involves the community in several aspects of the research design and emphasizes 
transformative actionable changes. Activist scholarship is a specific type of social justice 
scholarship that emphasizes active engagement with, advocacy for, and organizing with 
groups in society that are marginalized (Carragee & Frey, 2016). This could take the form 
of political organizing, protests, rallies, marches, campaigns, lobbying, direct action, civil 
disobedience, and so on. We will focus further on this type of work in the final chapter of 
this book. 


Why Do Numbers Have Power? 


In Communication, qualitative research methods—with the central position they reserve for 
issues of ideology, power, identity, and subjectivity—are frequent locations for social justice 
research. Yet, quantitative research is uniquely poised, as well, to effect change through the 
principles and practices of the scientific method. We have subtitled this book The Power of 
Numbers for Social Justice because we think quantitative methods available to researchers can be 
utilized to make strides toward a more just and equitable world. Importantly, many people are 
persuaded by numbers and statistical analysis (Allen & Preiss, 1997). This, in itself, shows the 
potential of quantitative research to raise awareness, spur concern, test proposed solutions, or 
otherwise address social justice issues and topics. In their book, Feminism Counts: Quantitative 
Methods and Researching Gender, Christina Hughes and Rachel Lara Cohen (2012) observe, 
“Quantification remains the ‘gold standard’ for much social science and policy-oriented 
research” (p. 2). This is mainly because numerical data are often assumed to be credible, reli- 
able, relatively unbiased, and verifiable. 
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Data for Black Lives is an excellent example of the power of numbers, http://d4bl.org. 
Here is what the group says under the About tab: 


Data for Black Lives is a movement of activists, organizers, and mathematicians com- 
mitted to the mission of using data science to create concrete and measurable change in 
the lives of Black people. Since the advent of computing, big data and algorithms have 
penetrated virtually every aspect of our social and economic lives. These new data sys- 
tems have tremendous potential to empower communities of color. Tools like statisti- 
cal modeling, data visualization, and crowd-sourcing, in the right hands, are 
powerful instruments for fighting bias, building progressive movements, and 
promoting civic engagement. 

(emphasis added). 


The potential persuasiveness of quantitative data also stems from the fact that it is derived 
through the scientific method, as we will describe further. The public generally trusts that 
scientific researchers will use public resources well and will provide knowledge to both inform 
public policy and work toward improvements in society (Resnik, 2009). In national public 
opinion data in the U.S., 71% of respondents reported believing that government investment 
in basic scientific research pays off, and 79% said that scientific research has made life easier for 
most people (Funk & Rainie, 2015). Although there is some indication that trust in statistics 
and research is eroding (Spiegelhalter, 2017), there remains an underlying openness to the 
power of numbers held broadly among the public that provides an opportunity for researchers 
inspired by a desire to challenge the status quo. 

For social justice research, in particular, quantitative analysis can be used to “bolster the 
impact of small sample qualitative findings by documenting the ‘representativeness’ of oppres- 
sive conditions in larger representative samples” (Ponterotto, Matthew, & Raughley, 2013, 
p. 48). In other words, one key advantage of bringing quantitative methods to bear on ques- 
tions pertaining to social justice is that, as you will read later, quantitative research tends 
to look for phenomena occurring in larger and broader patterns compared to qualitative 
research. This is an important opportunity to illuminate what many people are experiencing, 
a potentially powerful argument in a push for social justice and social change. 

All of these assumptions about numerical data and the scientific method open the door for 
quantitative research to be a powerful tool for social justice. On the other hand, “data” of any 
sort can also be used to vilify, dehumanize, and rationalize discrimination and/or mistreatment 
and can ultimately support rather than challenge racism, homophobia, classism, Islamophobia, 
sexism, and ableism. The very power of numbers that can potentially be harnessed as a force 
for good can be and has been a force for ill as well, as you will see in Chapter 2 on research 
ethics. Numbers generated and presented in a manner that ensures power will stay in the 
hands of the powerful have a particular force. Indeed, it is important for us to acknowledge 
that, for some, quantitative research has been and even continues to be seen as fundamentally 
and irreconcilably at odds with critical, activist, and social justice work. 

But we argue in this book that research that is both scientifically rigorous and socially con- 
scious also has the power to contribute meaningfully to both individual and social change. 
Users of quantitative methods who wish to put the “power of numbers” to good use for 
society must learn and internalize the principles and practices of both the scientific method 
and of social justice, developing and applying a sensibility used to design and carry out their 
work. This balance puts considerable pressure on the researcher, as is apparent in this quote 
from psychologists Kevin Cokley and Germine H. Awad: “Recognizing how quantitative 
methods have been used to legitimize social prejudices and perpetuate oppression, we have 
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oe) 


argued that the problem lies not with the methods themselves, but rather the users of the methods 
(Cokley & Awad, 2013, p. 36, emphasis added). We would point out that where there are 
potential “problems” with the users of quantitative methods, there are potential opportuni- 
ties, as well. 


Scholar Spotlight: Dr. Sut Jhally 


Figure 1.4 Scholar Spotlight: Dr. Sut Jhally 


Professor Sut Jhally is the Founder and Executive Director of the Media Education 
Foundation and Professor Emeritus in the Department of Communication at the Uni- 
versity of Massachusetts Amherst. Professor Jhally has written six books and numer- 
ous scholarly and popular press articles on a range of topics in Communication and 
Media and Cultural Studies. Yet, his main body of work takes the form of educa- 
tional videos. The Media Education Foundation (MEF) is a non-profit organization 
that “produces and distributes documentary films and other educational resources to 
inspire critical thinking about the social, political, and cultural impact of American mass 
media” (www.mediaed.org/about-mef/what-we-believe/). Founded in 1991, MEF has 
produced more than 70 films and videos on such topics as gender and culture, race and 
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representation, class and society, the culture of consumerism, the media and the Middle 
East, and politics and current events. 


Q: What are the ways in which research (as produced by academics or others) is utilized in 
creating the content of MEF films? Do you draw from research in making arguments, 
introducing issues, etc.? What role do you think research plays in fulfilling the mission 
of the MEE “to inspire critical thinking about the social, political, and cultural impact of 
American mass media”? 

Dr. Jhally: The main mission of the Media Education Foundation is to translate the 
latest cutting edge research produced in the academy into a form and language 
that non-specialist audiences can engage with. Hence MEF videos are drenched 
in the results of academic research and presented in a way that moves the dis- 
cussion forward in a productive, rational, and scientific way. Without academic 
research, MEF could not fulfill its mission. (The Mean World Syndrome, Joystick 
Warriors, Dysfunctional Societies are just three examples of research-based films.) 

Q: What do you think are the unique ways in which quantitative research in particular can 
contribute to your work at the MEF? Do you think numbers and statistics can help provide 
a foundation from which to spark a critical conversation? Why/How so? 

Dr. Jhally: Quantitative research is absolutely fundamental to how MEF films are pro- 
duced. For example, while qualitative research can tell us a great deal about how 
media images/messages do their semiotic work, and the possible ideological con- 
sequences of the interaction between media environments and audiences, [only] 
quantitative and empirical work can establish how pervasive the messages are and 
whether or not large numbers of people are changed because of exposure to them. 

Q: What do you think are the chief challenges or limitations associated with the successful use 
of quantitative research in your work? If numbers and statistics do play a role but are sup- 
plemented by other ways of introducing issues and providing evidence, why do you consider 
it necessary to supplement those numbers and statistics? 

Dr. Jhally: See my earlier response. While quantitative research can establish broad 
patterns in media content and media effects, qualitative work is good at thinking 
about the mechanisms as work in the production of meaning. MEF is methodo- 
logically agnostic when it comes to research and will use different kinds of studies 
to shed light on the always complex, and often contradictory, processes underway 
in whatever is being examined. 


The Rise of Quantitative Criticalism 


The possibility of using quantitative methods for critical research is increasingly being pursued 
in several fields such as public health, education, psychology, and other largely quantitatively 
driven research areas (Cokley & Awad, 2013; Else-Quest & Hyde, 2016; Garcia, Lopez, & 
Vélez, 2018; McLaren, 2017; Ponterotto et al., 2013). The term quantitative criticalist 
(Stage & Wells, 2014) has been advanced to encompass quantitative researchers who conduct 
research not to reveal some universal “truth” but rather to expose injustices and inequali- 
ties and bring data to bear on the structures and forces that help explain those injustices and 
inequalities. 
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Quantitative critical perspectives have shaped quantitative methods within critical race 
theory using “QuantCrit” approaches (Garcia et al., 2018), feminist methods (Hughes & 
Cohen, 2012; Sprague, 2016), and in indigenous contexts (Groh, 2018; Juutilainen, Jef- 
frey, & Stewart, 2019). Communication scholars such as Mohan Dutta (2010), a health 
communication researcher, and the second author of this book, a media effects researcher 
(Ramasubramanian & Banjo, 2020), have argued for combining critical approaches to quanti- 
tative social sciences for social justice. With this book, we join the growing number of social 
scientist voices within and outside Communication who say that quantitative researchers not 
only have something valuable to offer to the causes of social justice but also that they have a 
responsibility to think carefully about injustices and inequalities in the process of conducting 
their research. 

We share two important definitions of quantitative criticalism: 


e Counseling psychologists Ponterotto et al. (2013) suggest, “Criticalist researchers 
employ both quantitative and qualitative methods in establishing a dialectical stance 
with respect to the researcher-participant relationship that serves to empower the par- 
ticipants and stimulate the transformation of oppressive conditions to more equitable 
ones” (p. 44). 

e Public health scholar McLaren (2017) contends, “Quantitative criticalists use quantitative 
data and methods to pursue research questions that, for example, reveal inequities and the 
social or institutional factors that create and perpetuate them. Critical quantitative schol- 
arship may also question measures and analytic practices used in quantitative research, to 
ensure that they adequately represent circumstances and contexts, and do not themselves 
inadvertently perpetuate exclusion and hierarchy” (p. 391). 


We point out that in discussing why this type of research is conducted, both Ponterotto and 
colleagues and McLaren remind us that quantitative criticalist research is partly about the 
purpose of research. In bringing up issues of empowering research participants and thinking 
carefully about how to measure variables and analyze data, these scholars remind us, too, that 
quantitative criticalist work is also about being conscientious and deliberate about the pro- 
cess of research. Finally, in discussing how research can be used and toward what ends, these 
authors also remind us that quantitative criticalist work is about the outcomes associated with 
research. 

Decades ago, Communication scholar Mike Allen observed that “the quest for justice is not 
a fundamental logical component of the scientific process (as previous unethical investigations 
demonstrate). Scientists are guided as they may be by conscience and the desire to improve 
conditions, but the requirements of conducting a scientifically sound study say nothing about 
social justice. Thus, QSS (quantitative social science) unguided by a critical spirit poses many 
challenges and may be dangerous” (Allen, 1999, p. 374). Using this book as a guide, we hope 
that emerging Communication researchers will develop their skills at mastering the scientific 
method to conduct quantitative research at the same time that they develop and put into 
action a critical consciousness regarding equality, inclusion, and justice. 

Indeed, the latter element—the critical consciousness—may be the most vital factor in 
conducting quantitative research for social justice action. Psychologists John G. Ponterotto 
et al. (2013) contend that “the social justice anchor of the research study lies in the multi- 
cultural awareness and competence of the researcher rather than with a component of the 
research method. Without high levels of multicultural competence of the researcher, both 
quantitative and qualitative procedures run the risk of further marginalizing and oppressing 
minority groups” (p. 57). We agree. 
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Figure 1.5 Illustrating the relationships among science, advocacy, and practice 


Source: Emily Riewestahl 


In summary, then, we argue that bringing quantitative criticism to Communication involves 
three interrelated pursuits: 


e Choosing research topics that matter for social justice; 

e Developing a social justice consciousness and applying that consciousness to how the 
research is designed and carried out; 

¢ Attending to the important question of how, by whom, and for what purpose the knowl- 
edge that emerges from the research is used. 


Key Features of Quantitative Social Science Research 


Quantitative research in Communication applies the main features of the scientific method 
developed for the natural sciences for its investigation of communication-related phenomena. 
A quantitative research study follows a rather fixed set of steps—all of which will be outlined 
and discussed in great detail in this book (Figure 1.6). Each of the steps has its own possibilities 
and limits for researchers attempting to conduct their research from a critical and/or a social 
justice point of view. We will introduce the steps briefly here and return to each of them in 
the chapters to come. 


1. Typically, a quantitative research project begins with a question the researcher would like 
to try to answer. 

2. After identifying that question, the researcher finds out what is already known about the 
phenomenon they are studying through a review of the literature. The “literature” is 
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Figure 1.6 The scientific method 


Source: Wikipedia user named Whatiguana, https://commons.wikimedia.org/wiki/File:Scientific_Method_3 jpg 


the previously existing knowledge on the topic, usually found in the form of prior studies 
that the researcher can access to begin to understand the phenomenon of interest. 

Next, the researcher poses hypotheses that use the literature and theories from prior 
research to make predictions about what is expected to occur in the study at hand. 

The next step is to write a detailed methods section in which a roadmap for how the 
study will be conducted is provided alongside details about why each decision about how 
to proceed has been made. 

The next stage is data collection, and because we are talking about quantitative research, 
those data take the form of quantitative values (like a research participant’s response on 
a scale of 1 to 5). The quantitative data are subsequently analyzed using statistical pro- 
cedures that look for associations or differences in those values and are written up in a 
results section. 

Finally, in the discussion section, findings are discussed in terms of their implications 
for the phenomenon in question, their relationship to prior findings from the literature 
review, their relevance for society and social actors within, and the strengths and limita- 
tions of the study. Knowledge is generally regarded as produced incrementally, with a 
single study taking small steps toward revealing the contours and complexities of the 
phenomenon studied. In fact, quantitative studies often end with recommendations for 
the next steps to take in future research on the same topic. 
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The quantitative scientific method has a number of defining features that characterize its 
use as a tool in the production of knowledge, including for social justice purposes. The rest 
of this book will explore each feature of the scientific method in full. For now, however, 
we introduce three basic principles to help you begin to understand the underlying logic of 
quantitative research: 


e The principle of objectivity; 
e Aggregation of numerical data points; 
e Deductive reasoning (or a priori design). 


The Principle of Objectivity 


Quantitative research is concerned with minimizing the role that the researcher’s own feel- 
ings, priorities, experiences, and opinions play in shaping research results and the processes 
that produce them. Even post-positivists, who reject the idea of pure objectivity, will employ 
strategies to attempt to lessen the impact of their own subjectivity in the research. The under- 
lying objective is that the data produced by quantitative research should be able to be rep- 
licated by other researchers. If two researchers use the same research procedures, as long as 
what they are studying is fairly stable, they should observe, more or less, quite similar findings. 
This is because the observation of those findings was not based wholly on their own subjec- 
tive point of view. 

The principle of objectivity, even if it is only partially met, has some potential advantages 
for doing research toward social justice ends. Sometimes, people in general or policymakers in 
particular are more likely to be persuaded by research that is seen as wider than one research- 
er’s point of view. Using the scientific method, for example, the quantitative researcher is not 
typically viewed as engaging in selecting or “cherry-picking” observations or data for the 
purpose of making a particular argument. The scientific method’s central goal of producing 
verifiable evidence that is not attributed to the researcher’s own subjectivity has a long history 
of influencing policy and other forms of social, economic, and political change (Cokley & 
Awad, 2013). 

Yet, striving toward objectivity can also breed detachment from those researched, and in 
extreme cases, even foster dehumanization. It can also breed detachment, too, from the politi- 
cal, economic, social, and cultural conditions in which knowledge is produced and in which 
we all live our lives. Since much of the goal of social justice research is to raise up the voices 
of the oppressed, to give voice to their subjectivities and positionalities, having researchers 
somehow be removed from their own subjectivities and positionalities can be at odds with 
this goal. Many scholars have written, as well, about the manner in which an allegiance to 
objectivity reinforces a singular perspective that reflects and reinforces White male power 
(e.g., MacKinnon, 1983). 

Fundamentally, the development of what we have termed a “critical consciousness” does 
implicate the researcher’s own subjectivity, sense of what is just or unjust, and motivation to 
intervene. To do research with social justice goals at the forefront is not to stand dispassion- 
ately on the sidelines, and therefore there is a built-in ideological component that is at odds 
with a pure sense of objectivity that needs to be recognized and acknowledged. The way 
forward, therefore, is likely to be a delicate balancing act, in which dichotomies of objectivity 
and subjectivity are rejected in favor of a negotiation of one’s role in research and social action, 
one’s positionality in relation to the people or topics studied, and one’s interpretation of the 
data gathered through the scientific process. 
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Learning and applying the steps involved in the scientific method will help the researcher 
minimize unintentional bias in data collection and analysis on one side of the balancing act. 
Indeed, many of the components of the scientific method can be considered means of mini- 
mizing the impact of researcher subjectivity, including engaging in deductive reasoning to 
create and test hypotheses, producing samples with an eye toward representativeness, and 
reporting on aggregates rather than unique cases. Each of these conventions serves to reduce 
the role of researchers themselves unduly influencing the research process and the data pro- 
duced. The researcher’s own passions and priorities regarding what to study, how to study it, 
and whether and how to engage in social action regarding the issues at stake in the research 
on an intentional level is on the other side of the balancing act. 

The balancing act between objectivity and subjectivity among researchers who use the 
scientific method to produce knowledge was on display in September 2020. At that time, 
the editorial board of the Scientific American decided to endorse a U.S. presidential can- 
didate for the first time in its 175-year history. At that time, as well, a scathing editorial 
about U.S. President Donald Trump’s handling of the COVID-19 pandemic was published 
in the journal, Science. In an interview for Wired magazine following the editorial, Science 
editor H. Holden Thorp was quoted as saying, “You know, there is no apolitical science. 
Science is done by human beings in political environments funded by the federal govern- 
ment. The notion of apolitical science has never been real to begin with” (Rogers, 2020). 

In Communication, not all research is funded by the federal government. However, the 
first part of Thorp’s quote would certainly apply. In Communication, as in any field of study, 
“Science is done by human beings in political environments.” That simple but profound 
acknowledgment already begins the balancing act of objectivity and subjectivity. 

The question of objectivity in research was challenged even more powerfully in a June 2020 
editorial in Scientific American endorsed by 500 women scientists whose identities were not 
provided in order to protect them from any potential negative consequences for their careers 
regarding what they wrote. Here is what they had to say: 


If scientists don’t explicitly say #BlackLivesMatter, if they don’t speak up on justice and 
social issues, they risk overlooking solutions, discarding talent, and perpetuating toxic 
power dynamics. We must all hold ourselves and each other accountable to dismantle the 
systems of oppression that persist in our society, the very systems that claimed the lives of 
George Floyd and countless other Black people. That work should start by denouncing 
racial injustice and violence in no uncertain terms, but it cannot end there. . . . Addressing 
racism in science has to include fundamentally changing the scientific institutions them- 
selves, anything short of that is just lip service. Silence is never neutral. Neither is science. 

(500 Women Scientists Leadership, 2020) 


Research in Action 


Facebook made headlines in 2014 when it introduced a custom function in which U.S. users could 
specify their gender identity using about 50 different options (Figure 1.7a). One year later, Face- 
book changed the custom feature to allow users to type in any description of gender identity that 
users wanted to include, rather than trying to anticipate the full range of possibilities (Figure 1.7b). 
This is an excellent illustration of how difficult it can be to anticipate and fully account for the entire 
range of human experiences and identities. 


Foundations and Definitions 


: ; ( Fe | Salon by 
Basic Information |J Edit paris 
Gender | Custom + 
Gender $} Friends ~ 
Tral 
Trans Female 
Trans Male - 
Trans Man 
Trans Person 
i Trans Woman 
Birthday Trans* RY 
Trans* Female a~ 
Trans* Male 
Trans* Man 
Interested In . RY 
Figure 1.7a Gender categories 
Source: Screenshot from Facebook.com 
Basic Info 
© Add a language 
© Add your political views 
© Add who you're interested in 
Custom z 
Gender 
What pronoun do you use? 
v 


Female: "Wish her a happy birthday!" 


ê Only me 


Figure 1.76 Gender pronouns 


17 


18 Foundations and Definitions 
Aggregation of Numerical Data 


As we mentioned, in quantitative research, data take the form of numbers. So, rather than ask 
a research participant who is filling out a survey to use words to explain an opinion that they 
have, for instance, the quantitative researcher measures opinions (or anything else) numeri- 
cally. Measurement of opinions and other variables will be the topic of Chapter 3. But to 
illustrate this point for now, imagine that the research participant is given a statement that 
expresses an opinion and is asked to select a number that will represent more or less agreement 
with that statement. For example, a survey item might state, “Racism is the most important 
social issue in the United States,” and the person filling out the survey might circle an option 
from the following set of choices: 1 = strongly disagree, 2 = disagree, 3 = neither agree nor 
disagree, 4 = agree, and 5 = strongly agree. In this way, then, the data gathered in the research 
takes shape from the numerical values chosen by each person who participates. 

Typically (but not always) in quantitative research, data are drawn from the individual and 
then aggregated—combined to form groups of individual data points about whom/which 
conclusions are drawn. In using summary statistics, for example, individual data points 
are transformed into averages (like the mean response when you ask people to report their 
household income) or frequencies (like the percentage of the sample who said, “Yes, I use 
social media to follow politics regularly”). Rather than reporting on each individual’s answer 
to each individual question used to study something, we aggregate or combine those answers 
to simplify the data and find patterns. 

Aggregation is generally considered a necessary process in quantitative research, which is 
more often concerned with broad patterns and not necessarily with the details of each case 
of individual variation that occurs outside of broad patterns. In a quantitative study, it is often 
impractical and therefore unadvisable to provide the exact responses of each of the poten- 
tially hundreds of research participants. Therefore, we use summary statistics that are aggre- 
gated from individual responses to report averages in the sample or to report most common 
responses. All individuals are included in averages, of course, since averages themselves are 
derived from all of the data. Yet, by focusing on average responses, we run the risk of obscur- 
ing experiences, points of view, and responses outside of the norm. 

If one is concerned with voice, inclusion, and power relations, the use of the aggregate 
rather than the individual can reify dominant rather than marginal positions. In other words, 
if quantitative research focuses on the majority—through reporting, for instance, that 73% 
of students on a given college campus have not experienced discrimination—we may lose 
track of the 27% who have. Their experiences are not the majority but are likely to be the 
most important aspect of the phenomenon to bring to light through the research. Even in the 
process of aggregation, then, the quantitative researcher working to be responsive to issues 
and questions of social justice must remember that there are real people and real stakes behind 
those numbers. Researchers should remain open and curious about even non-majority posi- 
tions and points of view. We will return to this issue later in this book. 


Deductive Reasoning 


As we have introduced in this chapter, hypotheses are usually derived from applying the 
findings of prior studies and the principles of existing theories to the new set of circum- 
stances that an individual study explores. Then, the data are gathered to see whether those 
hypotheses find support, deducing from the data whether the prediction holds in the new 
circumstances. This is the principle of deductive reasoning that is at the heart of quan- 
titative research. In order to achieve this, the researcher has to anticipate the full range of 
responses that research participants will have to the conditions and materials (such as a 
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Figure 1.8 A cartoon about the concept of objectivity in research 


Source: From the comic strip ““Nestlings” by Warren Clements 


survey) that they will encounter in the study ahead of time and devise a way of assigning 
each potential response a number. This is known as an a priori design, with a priori mean- 
ing that the predictions of what will be observed in the study are made before the observa- 
tions themselves are collected. 

Have you ever taken a survey and thought to yourself, I can’t find a response option that accu- 
rately reflects how I feel about this topic? That is an issue with the a priori nature of quantitative 
research, in which the researcher has to anticipate and have a way of organizing everything 
that a survey respondent might want to report. Ifa survey respondent feels that their views are 
not represented among the options a researcher has decided to include for a survey question, 
the researcher has not adequately anticipated the full range of options that would correspond 
with the complexity of opinion or experience that exists. 

Failure to have a complete list of options that fully correspond to the entire range of pos- 
sibilities in what you are studying is not only faulty social science, it also has negative impli- 
cations for social justice. Individuals who do not see an option that corresponds with their 
Opinion, experience, sense of self, etc. have the potential to feel marginalized by the research, 
since the implication is that their opinion, experience, sense of self, etc. was not apparent 
or valued enough by the researcher to be included in the study’s design. In fact, sometimes 
quantitative researchers might include an option labeled “other” as a catch-all category in a 
survey so that they can be assured that they have anticipated all possibilities exhaustively ahead 
of collecting data. But, similarly, circling an “other” option can result in feeling excluded, 
undervalued, and pushed to the margins. 

Consider the complex matter of measuring the gender identity of research participants. 
Not to be conflated with biological sex (that is assigned at birth), nor with sexual orienta- 
tion or sexual identity (that correspond with romantic and/or sexual attraction), gender 
identity is a person’s view of the self that can reflect various positions on a continuum of 
societal and cultural definitions anchored by the concepts of masculinity and femininity, 
or the person may think outside of those definitions entirely. How might a person who 
is gender non-conforming, for instance, feel when completing a survey and only being 
presented with two options to report their gender, man and woman? How would they 
answer? To read more about this important issue, research by Gloria Fraser (2018) shares 
several examples of best practices for more inclusive language around gender-related 
measures in social sciences. 
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As you saw in the Research in Action feature of this chapter, Facebook made headlines in 
2014 when it introduced a custom function in which USS. users could specify their gender 
identity using about 50 different options. One year later, Facebook changed the custom fea- 
ture to allow users to type in any description of gender identity that users wanted to include, 
rather than trying to anticipate the full range of possibilities. This is an excellent illustration of 
how difficult it can be to anticipate and fully account for the entire range of human experi- 
ences and identities. 


Summary 


In this chapter we learned about key concepts within social justice scholarship and quantitative 
research methods. Research methods are techniques and approaches to produce knowledge. 
Social science research methods focus on producing knowledge that helps us understand the 
world around us. Basic social science research is motivated by researchers’ curiosity and inter- 
est in theorizing about the world. Applied social science research aims to address and solve 
real-world problems relating to the social world we live in. Social science research can help 
explore, describe, evaluate, explain, and transform the social world around us. There are four 
main approaches to social science research: quantitative, qualitative, mixed methods, and com- 
munity-based research. Quantitative research uses numerical data to test hypotheses, measure 
variables, and explain relationships. Qualitative research emphasizes more subjective experi- 
ences, “thick” or in-depth descriptions, and sense-making of the social world. Mixed methods 
approaches combine quantitative and qualitative methods. Community-based research has the 
community as the location or setting for research. These methods are situated within three 
main research paradigms: (post)positivism, constructivism, and critical approaches. 

Social justice research is a type of applied research within the critical paradigm whose goal 
is to attend to and address social inequalities, power imbalances, and oppression. There are 
many types of social justice research such as translational research, public scholarship, engaged 
scholarship, policy research, community-based participatory research, and activist scholarship. 
Communication, with its emphasis on practical theorizing and history of critical scholar- 
ship, provides a valuable perspective from which to approach social justice research. While 
qualitative research has long played a central role in Communication social justice scholarship, 
quantitative research is poised, as well, to inform such efforts. 

Quantitative methods for social justice use the “power of numbers” and the scientific 
method for social good to bring about meaningful and measurable social change. Quantitative 
criticalists use the power of numbers and quantitative research methods to expose and address 
social injustices and inequalities. They develop a critical consciousness that is used to question 
methods, measures, and practices of how research is conducted. Quantitative methods are 
rooted in the scientific method, which starts with reviewing the literature, posing hypotheses, 
determining the methods to test the hypotheses, collecting quantitative data, analyzing the 
data using statistical procedures, and discussing the implications and limitations of the findings. 
Quantitative research methods have three key features: the principle of objectivity, aggregation 
of numerical data points, and deductive reasoning. However, we have shown in this chapter 
how each of those features has nuances and complexities for social justice research. 


Discussion Questions 


1. What does social justice mean to you? What topics and issues motivate you to engage 
in research? What are some research questions that Communication scholars, especially 
quantitative criticalists interested in social justice, are well-positioned to address? 
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2. Do you agree with the “power of numbers” central argument put forth in this chapter? 
Do you think quantification, numbers, and data have a particular persuasive force among 
the public and among policymakers? Why or why not? 

3. What is your opinion about the role that objectivity can or should have in research? Does 
your opinion change if the research is social justice-oriented? 

4. What do you anticipate would be the greatest challenge of doing quantitative research for 
social justice? What do you think is the greatest opportunity? 
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2 Ethical Considerations 


Some research methods books include a chapter on ethics at the end of the book, which 
can make that chapter seem almost like an afterthought. For this book, we place this chapter 
on ethics at the beginning to emphasize its centrality in quantitative social justice research. 
We also conceive of ethics broadly in this chapter, as much more than a simple checklist for 
researchers to use to avoid harming research participants. Rather, we demonstrate that ethics 
is closely connected to reflexivity, thinking through our obligations to conceive and carry out 
research with sensitivity and integrity. 

The word ethics is derived from the Greek word ethos, which means character. It is closely 
related to concepts such as morality, integrity, and truthfulness. Morality is about making 
judgments about what is right and wrong. Integrity is about the alignment of actions and 
behaviors with one’s ethical and moral codes. As you will see in this chapter, ethics, morality, 
and integrity are essential to use as guiding forces in social science research, and they take on 
greater importance if you are working toward social justice in your research. 

The origins of ethical principles can be traced back to several Eastern philosophies. For 
instance, ahimsa (non-violence) and satya (truth) are core principles emphasized in the 
teachings of Buddhism, Jainism, and Patanjali’s Yogasutra around 400-500 BCE. Satyagraha 
or nonviolent civil resistance was popularized by Mahatma Gandhi (Figure 2.1) and later 
adopted by Martin Luther King, Jr. in the 20th century. These ancient ethical principles con- 
tinue to be relevant to modern-day research practices. Researchers in Communication and 
other social sciences are often motivated to shed light on “truth” and “reality” through their 
research inquiries. There is also a long tradition in social justice scholarship of conducting 
research that either implicitly or explicitly argues for non-violence, including reducing vio- 
lence that is structural and systemic in nature as well as that which is interpersonal. 

Another important historical marker in the codification of ethics is the Hippocratic Oath 
in Greece around 300 BCE, which was influenced by the Buddhist Physicians’ Oath called 
Vejjavatapada. The primary principle of this oath that has been adopted by biomedical 
researchers and practitioners is “Do no harm.” For social researchers, given contemporary 
practices put in place to protect research subjects, physical harm is, thankfully, less likely to 
occur. Instead, risks to participants in social science research tend to involve more psychologi- 
cal and emotional forms of harm rather than physical and physiological forms. Yet, those risks 
are critically important, as well, and therefore social scientists, too, should begin with the basic 
principle of avoiding or minimizing harm. 


The Significance of Ethics within Social Justice Research 


Ethical principles govern all aspects of the research process, including what is studied and 
who is involved in the process. As researchers interested in human communication, it is 
important to foreground ethical considerations to ensure that research does no harm and, in 
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Figure 2.1 Mahatma Gandhi 
Source: Elliott & Fry, https://commons.wikimedia.org/wiki/File:Mahatma-Gandhi,_studio,_1931 jpg 


fact, maximizes benefits to participants and to society at large. The value systems into which 
researchers are socialized shape research decisions and experiences. Intrinsic moral values such 
as honesty, justice, equity, caring, and fairness guide social justice scholarship. 

What is discovered through research is just as important as how it has been discovered. In 
other words, the means that lead to scientific discoveries are just as important as the discoveries 
themselves. Social justice-oriented social scientists typically choose to study topics and address 
research problems that align with these values. These intrinsic moral values influence what 
topics are studied, which methods and theories are included, who is cited in the research, 
who are the collaborators and participants, how interactions with research participants are 
structured, how the research is written up, how the research findings are disseminated, and 
how the impact of such research is gauged. 
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Indeed, ethics is embedded in every single aspect of decision making and practice of 
research, especially for social justice scholars. Consider the case of a strategic communica- 
tion researcher whose study passes all university regulatory compliance board requirements 
but aids in the marketing of unhealthy and addictive foods to vulnerable populations. Or a 
scenario in which researchers mine personal data from social networking sites in ways that 
compromise private financial information, intimate conversations, and personal opinions. Or 
when an intercultural communication scholar’s work is co-opted for use in military operations 
that result in the loss of lives. These are all examples of how the ethics of the research takes 
on a different dimension in particular communicative situations. In this chapter, we can, at 
best, merely scrape the surface of this complex topic by drawing attention to a few essential 
guiding principles and providing direction on practical tips for the design and implementation 
of research in ethical ways. 

There are many reasons why researchers need to be attentive to ethical issues. Israel (2015) 
puts forth some motivations: to protect research participants, to maintain trust so that the 
research process can continue, to increase research integrity, to be in compliance with pro- 
fessional codes of conduct, and to respond to new ethical research situations (such as online 
and digital participants). For example, within communication contexts, the emergence of 
online media and social media sites such as Twitter and Facebook is challenging the defini- 
tion of “public spaces” and “research participants,” further complicating how Communication 
scholars approach research in ethically responsive ways in a dynamic, mediated, socio-political 
environment. 

Ethical misconduct can jeopardize the professional reputation of scholars and can result in 
withdrawal of funding, restriction of access to grants, and withdrawal of support for specific 
individuals or institutions. Although discourses about research ethics in quantitative meth- 
ods revolve around issues of misconduct, falsification, deliberate fraud, accidental or 


Figure 2.2 Ethics, accountability, principles, integrity, and values are closely related 
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intentional oversight, and fabrication of data, these topics are not the only areas of 
emphasis of this chapter. As social justice scholars, we need to push past the assumption that 
scholars can consider themselves ethical merely for not doing anything morally outrageous 
that could be considered academic misconduct or malpractice. While these violations of ethi- 
cal principles are important, it is crucial to focus our discussions on the more complex and 
less straightforward issues where the answers are not clear cut or obvious. 

In this chapter, we raise important ethical questions for quantitative social justice research- 
ers in Communication in designing, implementing, and disseminating one’s work. Some 
questions we will discuss are: What ethical considerations arise in conducting quantitative 
Communication research? What particular ethical considerations stem from quantitative 
social justice research within Communication research? How can researchers navigate these 
ethical questions and challenges? In this chapter, we apply ethical considerations and provide 
practical resources to help facilitate your development as ethical, quantitative, and social jus- 
tice researchers. 


History of Ethical Violations in Scientific Research 


Unfortunately, some forms of quantitative research have a history regarding social justice that 
is spotty at best and sordid at worst. In 1946, the infamous Nuremberg trials (Figure 2.3) 
brought to light the horrific medical experiments conducted by Nazi scientists. This history 
of research abuses especially during World War II led to the development of the Nuremberg 
Code in 1949, emphasizing the need for voluntary participation of research participants. 
Beyond war times, unethical research violations continued. 

One of the most egregious examples of research using quantitative methods that severely set 
back rather than advanced social justice is the Tuskegee Syphilis Study. Over the course of 
a 40-year period that began in 1932 (well after the health dangers were known and well after 
treatment options were available), the United States Public Health Service studied the effects 
of untreated syphilis among African American men (Figure 2.4). Participants were offered 
only very modest incentives for participating, and the researchers did not ensure that they 
knew what to expect, especially regarding the fact that they would be studied but not treated 
in the research (Jones, 1981). A rhetorical analysis of the medical reports generated from the 
research came to the conclusion that the principles of objectivity and detachment between the 
researcher and those researched within the scientific method facilitated the racism and dehu- 
manization of suffering that allowed for that study to continue for so many years (Solomon, 
1985). Observing the deterioration of the health of research participants without helping and 
extracting data about suffering without the participants’ full knowledge and/or consent in the 
name of science is an unambiguously harmful use of quantitative methods. 

Other examples are the infamous Milgram experiments and the Stanford prison experi- 
ments. The Milgram experiments were conducted in the 1960s by Stanley Milgram at Yale 
University. In the studies, electrical shock was used to test how far participants would go in 
terms of obeying orders to harm others. These experiments caused concerns because they 
caused severe psychological distress to participants. They also used deception and did not 
allow participants the option of withdrawing from the study. In the 1970s, the Stanford prison 
experiments conducted by Zimbargo and colleagues simulated a mock prison in the basement 
of the psychology department by assigning participants to being either guards or prisoners. 
The participants assigned the role of prisoners were mistreated for a two-week period and 
experienced severe stress, depression, and anxiety. 

An example of the use of quantitative logic and data to arrive at conclusions that are 
antithetical to social justice that is more guileful can be found in the book, The Bell Curve: 
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Figure 2.3 Proceedings at the infamous Nuremberg Trials in Nazi Germany 


Source: United States Army, https://commons.wikimedia.org/wiki/File:US_Army_51610_The_Case_Against_the 
Defendents.jpg 


Intelligence and Class Structure in American Life by Richard J. Herrnstein and Charles Murray 
(1994). Herrnstein and Murray used quantitative analyses to make claims about the role of 
intelligence in explaining social circumstances and economic struggle. They concluded that 
many experiences of inequality are explained by differential IQs which are, in turn, largely 
inherited, based in part on race, and therefore mainly unchangeable. 


Belmont Report and Institutional Review Boards 


The ethical violations and atrocities such as the first few examples just described led to the 
establishment of the National Commission for the Protection of Human Subjects and the 
Belmont Report (National Commission for the Protection of Human Subjects of Bio- 
medical and Behavioral Research, 1979) to prioritize the rights of human participants. The 
central principle is that research participants be viewed as people first and then as research 
participants. 

Based on the Belmont Report, Institutional Review Boards (IRBs) were set up across 
the United States, including at colleges and universities given that they typically employ 
researchers. IRBs review protocols and plans for research to ensure those protocols and plans 
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Figure 2.4 Tuskegee syphilis experiment 


Source: National Archives Atlanta, GA (U.S. government) https://commons.wikimedia.org/wiki/File: Tuskegee-syphilis- 
study_doctor-injecting-subject.jpg 


meet ethical requirements, and they do so before that research is allowed to be conducted. 
They continue to be an important part of social science research ethics. They typically include 
a minimum of five members including at least one scientist and another non-scientist to 
ensure that ethical standards are being applied. Prior to conducting a study or contacting 
research participants, researchers need to apply for IRB review. As the first step, all researchers 
need to complete ethics training every few years in order to be certified to conduct research 
with human participants. Such training is often available online at several institutions of higher 
education. 

The institutional ethics review boards and related ethics training focus on three main princi- 
ples: respect, justice, and beneficence, which were established as part of the Belmont Report. 
Respect is about treating research participants as autonomous agents and taking steps to pro- 
tect the rights of those with diminished autonomy. Justice focuses on making sure that the 
risks and benefits of research are distributed fairly to all participants. Beneficence emphasizes 
the ethical principle of ensuring that harm is minimized and benefits are maximized in the 
research process. These principles are enacted into practice through review of IRB application 
forms that typically include the researcher reporting information about the research purpose, 
the process used to inform participants about what the research will entail and to obtain their 
consent to participate, a description of recruitment processes, provisions for confidentiality, 
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analysis of possible risks versus benefits of the study, consideration of participants’ rights, and 
compensation to participants (if any). 

In line with the Belmont Report, participation in research studies should be voluntary 
and not involve any coercion. Participants should have the right to withdraw from the 
study for any reason during the course of the research project without fear of repercussions 
or rebuke. Let’s say that you are conducting a study on the prevention of cyberbullying with 
middle school children, using the school as the setting, and participation in the study leads to 
extra credit for the students. You should clearly assure the children that participating in the 
study is optional and that there will be no negative consequences such as losing grade points 
or disappointing their teachers for those who choose not to participate. It is important to also 
let the kids know that even if they have already started participating in the study, they can stop 
at any time without penalty. An alternative activity such as writing an essay on cyberbullying 
prevention should be made available to non-participants so that they can still earn the extra 
credit even if they decide not to participate. In taking these steps, the researcher is making sure 
not to be coercive, even if the coercion is unintentional. 

Obtaining informed consent from participants is important from a participant per- 
spective as well as for researchers. Informed consent is the idea that potential par- 
ticipants should be made aware of what they will be asked to do if they choose to 
participate in the research and can then make up their own minds about participation. 
Informed consent can be communicated and obtained through face-to-face meetings, 
email, video chats, social media, phone, websites, blogs, and other mediated technolo- 
gies that are relevant and culturally appropriate for the population that is being studied. 
Some groups such as minors below the age of 18 years are considered to be unable to 
give informed consent (such as the school children in the cyberbullying study example 
we mentioned). In these cases, a parent or legal guardian must give consent for the 
young people to participate in the study, and the young people themselves should agree 
to participate, as well, giving what is called their assent. 

Even if participants have provided initial written consent to participation, researchers should 
pay attention to process consent. At every stage of the research process, researchers should 
check in with participants to review consent issues to ensure that the participants understand 
that their participation is voluntary, that they have the right to withdraw from the study, and 
to ensure that they are not experiencing any harm or stress from the research process. Process 
consent is especially important to consider in studies that are longitudinal in nature—meaning 
that they take place over time—and involve multiple steps to complete. For instance, if we 
are interested in studying how victim-blaming on college campuses might affect the health 
and academic success of sexual violence survivors throughout their college education, such 
potentially sensitive topics will likely involve collecting data over multiple years. At every step 
of the research study, it would be wise for the researcher to check in with the participants to 
ensure that they continue to be okay with participating in the research. 

If the topic is particularly sensitive, then it is possible that even obtaining the signature 
of participants might put them in some danger. Under such cases, researchers can request a 
waiver of informed consent as an exception to the usual need to get a signed indication 
of consent. For instance, let us say that you want to conduct a survey on how college stu- 
dent organizations devoted to LGBTQ+ rights on campuses use communication strategies 
to spread awareness about their causes. However, it is possible that some LGBTQ+ students 
might not want to be identified even through signatures on the written consent form. Under 
these circumstances, researchers can request other forms of consent such as verbal consent 
rather than written consent. 
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In the following text we provide you with a sample informed consent form. This is a fairly 
standard template likely to reflect some of the common components of a consent form. Some 
sections may not be applicable for all types of research. This form can and should be modified 
for your specific research project. Note that the informed consent form should be written in 
simple language that is readable and easily understood by someone with an 8th-grade reading 
level. When working with participants in multilingual contexts, it is also important to make 
the informed consent form available in multiple languages. 

You will notice, as well, that the standard form has a place for the research participant 
to sign, indicating their voluntary decision to take part in the study. As we just mentioned, 
under some circumstances, however, you may not need to have research participants sign their 
names, since doing so involves giving up their anonymity. In some cases, then, pending IRB 
approval, you may be able to replace the signature part of the consent form with a statement 
such as “By continuing on to complete this survey, you are indicating your voluntary consent 
to participate.” 


Research in Action 


Sample Informed Consent Form 
Title of Research Study: 


Name of Investigators: 


Why are you being invited to take part in a research study? 


[Examples: You are invited because you are attending the online dialogue event. You are 
invited because you are enrolled in COMM 101: Introduction to Communication. You 
are invited because you are a member of the Black Students’ Council.] 


What should you know about a research study? 


e Someone will explain this research study to you. 

e Whether or not you take part is up to you. 

e You can choose not to take part. 

e You can agree to take part and later change your mind. 

e Your decision will not be held against you. 

e You can ask all the questions you want before you decide. 


Why is this research being done? 


[Insert the purpose here. Here is an example: The purpose of this project is to examine 
experiences pertaining to media use as it relates to various identities. We seek to gain 
an understanding of perceptions regarding the modern media environment via survey 
questions regarding media use and views regarding social identity.] 
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How long will the research last? 


You will be in this research study for about [insert duration of study]. 


How many people will be studied? 


We expect to enroll about [insert sample size] people in this research study. 


What happens if I say, “Yes, I want to be in this research”? 


With agreeing to this consent form, you agree to participate in a [insert duration]-long 
study [insert location: such as via phone, video chat, online survey, etc.]. [Share details 
about the procedure. For example, “You will complete an anonymous feedback form,” 
“You will participate in a focus group discussion,” or “You are agreeing that the session 
will be audio-recorded.”] 


What happens if I do not want to be in this research? 


[Give details of alternatives: For example: You can leave the research at any time with- 
out penalties. You can complete an alternative writing assignment instead of participat- 
ing in the study.] 


What happens if I say “Yes,” but I change my mind later? 


You can leave the research at any time and it will not be held against you. If you decide 
to withdraw from the study, please contact the investigator to grant permission regard- 
ing the use of data collected to the point of withdrawal. 


Will I be paid to be in this study? 


99 66 


[Insert information: “No, you will not be paid to be in this study,’ “If you participate in 
the online survey, you have the option of entering a raffle for a gift card worth $25. Two 
such cards are available in the raffle,” “During the focus group discussion, you will be 
provided with a pizza meal and soft drinks,” or “You will not be paid to be in this study 
but can earn extra credit points of 1% of our course in COMM 101 if you participate 
in this study.] 


Are there any potential risks to me in participating in this study? 


[Explain potential risks. For example: “There are no potential risks associated with your 
participation in this study. If at any point you do not feel comfortable continuing, you 
may refuse or discontinue the survey at any time,’ “The things that you will be doing 
have no more risk than you would come across in everyday life,’ or “Although the 
researchers have tried to avoid risks, you may feel that some questions/procedures that 
are asked of you will be stressful or upsetting. You do not have to answer anything you 
do not want to.’ 
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Will being in this study help me in any way? 


[Insert details about any benefits. For example: “We cannot promise any benefits 
to you or others from your taking part in this research,” “Your participation in this 
study may help to inform you on how to use online technologies for facilitated 
conversations, and you will learn about some resources related to COVID-19,” or 
“You will have the opportunity to learn more about critical perspectives on media 
studies.’] 


Will information from this study be kept private? 


[Insert information: For example: “Study information will be kept confidential. We will 
delete all identifying information. We will protect all participants’ identities by using 
pseudonyms.” 


Will audio or video recordings of me be made during this study? 


[Explain your answer. For example: “The researchers will make an audio recording 
during the study so that the conversation can be transcribed for analysis. If you do not 
give permission for the audio recording to be obtained, you cannot participate in this 
study.” ] 


I give my permission for audio recordings to be made of me during 
my participation in this research study. 

I do not give my permission for audio recordings to be made of me 
during my participation in this research study. 


Whom can I talk to in case of questions? 


If you have questions, concerns, or complaints, or think the research has hurt you, 
please contact [name of researcher] by email at [insert email address]. This research has 
been reviewed and approved by the Institutional Review Board (IRB). You may talk to 
them at [insert phone number] or by email at [insert email], if: 


e You cannot reach the research team. 

e The research team does not answer your questions, concerns, or complaints. 
e You want to talk to someone besides the research team. 

e You have questions about your rights as a research participant. 

e You want to get information or provide input about this research. 


Statement of Consent 


I agree to be in this study and know that I am not giving up any legal rights by signing 
this form. The procedures, risks, and benefits have been explained to me, and my ques- 
tions have been answered. I know that new information about this research study will 
be provided to me as it becomes available and that the researcher will tell me if I must 
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be removed from the study. I can ask more questions if I want. A copy of this entire, 
signed consent form will be given to me. 
Participant’s Signature Date 


Printed Name Date 


Investigator’s Affidavit 


Either I have or my agent has carefully explained to the participant the nature of the 
above project. I hereby certify that to the best of my knowledge the person who signed 
this consent form was informed of the nature, demands, benefits, and risks involved in 
his/her participation. 


Signature of Investigator Date 


Printed Name Date 


Types of Ethical Research Guidelines 


Ethical principles within Communication are guided by legal regulations, professional codes, 
and researchers’ personal values. Procedural ethics are legal regulations and policies that 
govern the ethics of research within a particular institution, state, or nation. For example, in 
some countries, it could be legally required for researchers to disclose any sexual violence or 
harassment information that is revealed to them. Professional ethics are ethical guidelines 
and principles set within professional organizations, which for Communication scholars could 
include organizations such as the International Communication Association or the National 
Communication Association. Personal ethics are the researcher’s own set of moral values 
and an ethical compass that guide their research, such as an ethics of care or a commitment 
to fairness, that may or may not always overlap with laws, institutional ethical policies, and 
professional ethical guidelines. We will elaborate on each of these in the following text. 


Procedural Ethics 


Legal regulations include any institutional, state, national, or international laws and policies 
that govern research ethics. For instance, Glass and colleagues (2018) point to how research 
ethics review boards can sometimes take a very narrow view on what ethics means. In this 
minimalistic view of research ethics, the focus is simply on “avoiding harm” to participants. 
Guillemin and Gilliam (2004) call this procedural ethics, where research ethics is viewed 
mainly as bureaucratic paperwork to be completed by researchers. Such forms of procedural 
ethics tend to be legalistic and documentation-orientated. It is important for social justice 
scholars to pay greater attention to ethics and integrity beyond compliance with institutional 
review boards (IRBs). Taking the IRB process as a vitally important opportunity—or even 
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obligation—to think carefully about the possible experiences of research participants, the 
potential intrusiveness of the research process, and the various ways that the new knowledge 
gained in the research can be used for good or for ill is a necessary starting place for any 
researcher, and it is especially critical when the research is specifically rooted in social justice. 


Professional Ethics 


Many social science disciplines have their own professional code of ethics. Within 
Communication, there are numerous professional organizations such as the International 
Communication Association (ICA), Association for Education in Journalism and Mass Com- 
munication (AEJMC), National Communication Association (NCA), Rhetoric Society of 
America (RSA), the International Association for Media and Communication Research 
(IAMCR), and the Association of Internet Researchers (AoIR). Not all of them have a pro- 
fessional code of ethics explicitly stated. Those that do have ethical codes vary greatly in the 
depth and breadth of issues covered. 

Generally, professional codes of ethics within Communication research discuss issues such 
as plagiarism, protection of research participants from harm, integrity in data collection and 
analysis, accurate and objective reporting of research, fair authorship and mentoring practices, 
conflict of interest, and confidentiality and fairness in reviewing research manuscripts for pub- 
lication in academic journals. Some associations also discuss professional codes of conduct for 
interpersonal communication, especially at conferences, such as being respectful, considerate, 
inclusive, and courteous. Organizations such as RSA not only go into detail about various 
offensive, harassing, disruptive, and inappropriate behaviors that will not be tolerated at their 
conferences but also discuss how such behaviors should be reported and what the conse- 
quences are for unacceptable behavior, which include banishment from future conferences. 

As the profession develops, new situations and research challenges arise which need to 
be addressed. For example, NCA’s code of ethics was adopted in 1963 (when it was still 
called the Speech Association of America) and reaffirmed in 2011 and edited again in 2017. 
Their current “Credo for Free and Responsible Communication in a Democratic Society” 
endorses freedom of speech, harm reduction, maximization of good, respect for precision 
and accuracy, and evidence-based reasoning (National Communication Association, 2017). 
ICA’s updated ethical code includes ethical considerations for social media use. It aims to cre- 
ate more awareness about the citation of unfinished works-in-progress, privacy breaches, and 
politically motivated personal attacks on researchers via social media. It suggests that audio 
and visual recordings of conference proceedings not be made without the explicit consent 
of all panelists and seminar speakers, that live-tweeting and use of the ICA hashtag does not 
include unprofessional language, and other specific guidelines for fair quotation of research 
through social media. 


Personal Ethics 


Research ethics is also influenced by intrinsic values and the researcher’s moral compass. 
Researchers who have centered social justice within their scholarship have developed more 
personalized and aspirational codes of ethics that go beyond satisfying IRB requirements. 
A commitment to a more just university and society at large, then, translates into examining 
the core principles of how research ethics itself is institutionalized within structures in research 
organizations. 

For instance, Maria K.E. Lahman (2018) calls for researchers to go from procedural ethics 
that focus on minimalistic codes of ethics to aspirational ethics that stem from the researcher’ 
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personal aspirational ethics standards, over and above those of the required procedures. 
Aspirational ethics emphasizes the highest ethical standards that the researcher tries to achieve 
rather than fulfilling the minimal ethical requirements. While regulatory or compliance-ori- 
ented ethics are governed by questions of legality, it is important to remember that legality 
is usually determined by power and authority rather than integrity and social justice. In fact, 
slavery was legal at some point in time, just as apartheid and the Holocaust were all lawful acts 
within the contexts of the nations where they were practiced. Therefore, researchers should 
be guided, as well, by their own inner moral compass even if not required by their legal regu- 
latory boards or professional organizations. 


Ethics and Social Justice Research 


Ethical dilemmas in social justice activist scholarship are often difficult to anticipate, com- 
plex, and culturally rooted. As individuals who are committed to dismantling oppression and 
making the world more peaceful, just, and sustainable, it is important for researchers to act 
with sensitivity, respect, and honesty in making our contributions to scholarship. This means 
working with integrity by being careful, responsible, and transparent about every step in the 
research process: from setting the research agenda to designing the research, collecting data, 
analyzing data, and disseminating findings. 


Figure 2.5 Social justice movements have deeply influenced research ethics 


Source: U.S. Information Agency, Press and Publications Service (ca. 1953—-ca. 1978) https://commons.wikimedia.org/ 
wiki/File:March_on_washington_Aug_28_1963.jpg 
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The social justice movements of the 1960s and 1970s (such as the civil rights movement, 
feminist movement, and LGBTQ+ rights movement; Figure 2.6) played a crucial role in 
shaping values and bringing about significant changes in scholarship and research. Critical 
communication scholars started not only paying increasing attention to inequalities in private 
and public spheres, in mediated and non-mediated spaces, and in local and global contexts, 
but they also started examining the ways in which such historical inequalities manifest within 
academia as well. This self-reflexive examination of power within the discipline and social 
sciences as a whole meant greater awareness about the ways in which researchers collude 
with and reproduce social hierarchies of oppression such as racism, sexism, homophobia, 
xenophobia, and ableism. These considerations vastly shaped research priorities and agendas 
within the field, which were further facilitated and solidified in the last decade or so within 
Communication. 

There was a reexamination of topics such as the role of power, agency, structure, and ideol- 
ogy, leading to the rise of critical studies in various disciplines, including Communication. 
Feminist scholars such as Carol Gilligan and Annette Baier distinguished normative ethics 
from what they framed as the ethics of care. Normative ethics emphasizes more abstract 
reasoning such as individual rights, universal values, and utilitarian approaches. In contrast, 
the ethics of care focus on maintaining nurturing relationships, emotional responses, and the 
importance of context. Incorporating an ethics of care means acknowledging that situational 
ethics often depend on affective “gut” reactions such as compassion and sympathy rather than 
calculated, rational, moral decision making. For instance, perhaps a Communication scholar 
who is studying the #metoo movement on Twitter notices that those who are sharing stories 
of sexual harassment are being constantly trolled, attacked, and even blamed for the violence 
they have experienced. Although it is not part of the research study, they might file a com- 
plaint or report hateful messages to authorities. 


Self-Reflexivity and Cultural Responsiveness in Social Justice Research 


Quantitative ethical frameworks largely stem from biomedical and physical scientific 
approaches, which assume relatively objective and “neutral” stances of the researcher and 
prioritize objectivity, rationality, truth, and neutrality. The relationship between quantitative 
researchers and their participants typically characterizes the researcher as a neutral observer 
who maintains a distance from the participants in order to protect the integrity of the data. 
Such values might seem to be at odds with social justice scholarship that is explicitly political 
and therefore not value-neutral. 

As we have seen in Chapter 1, for quantitative criticalists (Stage & Wells, 2014), social 
justice research is neither neutral nor apolitical since it explicitly aims to break down oppres- 
sive structures of inequalities in society. Rather, quantitative criticalists are committed to 
and invested in using the power of numbers to bring light to important social issues. It is 
vital that in doing so, researchers see themselves not as experts with superior knowledge to 
impart to others but as partners with participants and other stakeholders committed to using 
research to minimize social inequalities and maximize the benefits to marginalized groups in 
society. Researchers should avoid “othering” of marginalized groups, disregard for alterna- 
tive perspectives, and a lack of cultural competency in designing and implementing quanti- 
tative research studies. They need to start by acknowledging that knowledge, interventions, 
and methods are not value-neutral and rather that they reflect power structures and interests 
within society. 
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Lahman (2018) advocates for a culturally responsive approach to research ethics that incor- 
porates cultural responsiveness, relational ethics, and self-reflexivity. Cultural responsive- 
ness means that researchers understand the dynamic nature of the shared meanings, values, 
and beliefs of the groups and topics they study within their specific historical, political, social, 
and economic contexts. They value, validate, affirm, and sensitively accommodate the world- 
views of their research participants. They constantly examine their cultural assumptions in the 
research process. Relational ethics, which draws from feminist ethical approaches, fosters 
mutual respect and dignity between researchers and participants. Reflexivity is a critical self- 
awareness about how the researcher’s cultural values, socio-economic background, political 
orientation, assumptions, and worldviews might impact the research process. A researcher 
who is reflexive pays careful thought and attention to how their own roles and actions in the 
research process might enhance or inhibit ethical knowledge creation. 

When conducting research with under-resourced and underrepresented groups, research- 
ers should approach them with cultural humility rather than a sense of arrogance or cul- 
tural superiority, especially if they come from privileged backgrounds. It is important to 
remember that Communication researchers are a diverse group, as are the research partici- 
pants they study. Social justice researchers are often a part of the marginalized communities 
they study. Not all research projects on social justice focus on under-resourced groups, either. 
They can also study those in powerful positions of authority that lead to social inequalities 
and injustices. 


Figure 2.6 Communication as listening 


Source: Photo by Cliff Booth from Pexels 
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Figure 2.7 Scholar Spotlight: Dr. Mohan Dutta 


Scholar Spotlight: Dr. Mohan Dutta 


Dr. Mohan J. Dutta is Dean’s Chair Professor of Communication, and Director, Center 
for Culture-Centered Approach to Research and Evaluation (CARE) at Massey Uni- 
versity. He served as the Head of Department of Communications and New Media 
at the National University of Singapore, where he led curriculum revision, industry 
engagement, and a strategic footprint for communication studies in Asia, and Courtesy 
Professor of Communication at Purdue University. At National University of Singapore 
(NUS), he was the Founding Director of the Center for Culture-Centered Approach to 
Research and Evaluation (CARE), directing research on culturally centered, commu- 
nity-based projects of social change. The following section is re-posted with permission 
from Dr. Dutta’s (2017) blogpost on “Teaching our students the ethics of the heart” 
accessible at https://culture-centered.blogspot.com. 

The pedagogy of communication in the backdrop of the dramatic inequalities we 
witness across the globe has to be grounded in an ethic of the heart. How to build 
communicative practices that embody this ethic of the heart is the key question for 
culture-centered scholars working on social justice projects from/with the margins. 

The journey of culture-centered pedagogy begins with this realization: to address 
the global challenges we are in the midst of, communication has to be placed at the 
center—not the forms of communication that simply serve as the loudspeakers for 
the 1%, but communication that is fundamentally transformative. Such transforma- 
tive practices of communication begin by turning toward communities at the margins 
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as entry points to discursive sites, discursive processes, and discursive articulations— 
communication that begins not from the predetermined agendas of those in power, but 
instead from the very margins that are reproduced by the consolidation of power in the 
hands of the global elite. 

The basis of this renewed emphasis on communication is grounded in a commitment 
to developing practices and processes that seek to build infrastructures for communica- 
tion where the poor participate, enact their agency in deliberating over decisions and 
reasons that are meaningful to them, and find avenues for these decisions to be enacted. 
The pedagogy of the culture-centered approach grapples with this challenge of how 
best to build infrastructures of communication that open up discursive sites to the agen- 
tic possibilities from the margins. 

An ethic of the heart works through the concepts of friendship and listening as 
anchors to learning to do relationships, dialogues, and participatory practices, attentive 
to the margins that are (re)produced by contemporary social systems and seeking to 
undo the social structures that reproduce marginalization. 

To work from the concept of friendship is to first and foremost change the very 
basis of the relationship with the margins, from one of unequal power to one that seeks 
equality in the relationship, deeply aware of the inequalities that are fundamental to 
the nature of relationships between academic-activists and communities at the margins. 
An ethic of friendship seeks to understand through conversations with the margins the 
practices that produce the margins from the standpoint of those at the margins, reflect 
upon the productions of privilege through these conversations, and participate in com- 
municative processes to change the inequalities that produce the margins in solidarity 
with communities at the margins. To be in solidarity is to lend one’s body to the strug- 
gles of the margins, placing the body amid the risks and challenges that come with 
communication that seeks to transform structures. 

The concept of communication as friendship with the margins works alongside the 
concept of communication as listening. The work of the academic-activist as a 
listener foregrounds the role of communication in building theories from the margins 
through collaborations at the margins. The struggles of the margins amid the global dis- 
semination of the ideology of neoliberalism as the basis for global organization disrupts 
the pseudoscience of neoliberal claims-making. Listening thus becomes the basis for 
challenging and transforming the theories disseminated in global structures of neoliberal 
governance. The unfounded bases of the claims that form the foundations of the free 
market ideology are disrupted through stories from the margins, building evidence that 
“makes impure” the dominant knowledge structures. 

For more information visit the CARE website: www.massey.ac.nz/~wwceare/ 


Avoiding Exclusion, Tokenism, and Deficit-Based Thinking 


As we will discuss further in Chapter 4, inclusive research samples are one way in which 
voices of underrepresented groups are prioritized within the research process. Here, research- 
ers seek out samples beyond majority group populations who are easy to access for research 
studies. For instance, while conducting an experiment on how to design a message for a 
health campaign on heart disease prevention within Singapore, this could mean making sure 
that input and feedback are received from a wide range of potential audiences. Instead of 
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simply focusing on educated middle-class heterosexual working families, these groups could 
include ethnic Tamils and Malays, immigrant laborers, domestic workers, the elderly, and 
those from LGBTQ+ groups, all of whom might typically not be included in such a study. 

Within Communication, groups such as those from economically exploited social classes, 
people with disabilities, transgender individuals, religious minorities, indigenous communi- 
ties, and those from the Global South continue to be underrepresented within research pro- 
jects. Therefore, researchers should consider identifying and including voices such as these 
that are not typically centered within traditional research. By being more mindful of which 
populations are being included in the research process, theoretical perspectives are broadened 
by asking completely new research questions that would have otherwise not been considered. 
However, it is also important to avoid tokenism. Tokens are a small number of minoritized 
group members who are employed in positions traditionally occupied by dominant group 
members (Jackson, Thoits, & Taylor, 1995; Kanter, 1977). They are treated as representatives 
of the group that they belong to rather than as individuals in their own right. Tokenism 
involves small symbolic acts to appear to be inclusive without a serious commitment to it. 
Adding a female presenter to an all-male opening plenary session conference at the last minute 
is an example of tokenism. 

The dominant modes of theorizing and conducting research within Communication are 
themselves shaped by dominant Euro-centric Anglo-centric male perspectives. These cul- 
tural perspectives shape the research paradigms, structures, and interpretations from research 
questions, theories, methods, and dissemination of research. Researchers should move from 
deficit thinking to asset-based thinking when it comes to working with research par- 
ticipants, especially when those from privileged backgrounds are conducting research within 
underserved communities (Alper, Katz, & Clark, 2016). Deficit-based thinking frames minor- 
itized groups as lacking and inferior compared to dominant cultural groups. Communication 
research and theories might be biased in how they frame and represent these groups as they 
are often developed within Western Eurocentric patriarchal contexts. Within deficit ways of 
thinking, inequalities are often implicitly blamed on the marginalized groups due to perceived 
personal individual deficits rather than structural social factors. 

While working with marginalized groups, it is important to acknowledge and incorporate 
their cultural beliefs, perspectives, and experiences into the research process. One common 
way in which deficit thinking works is to blame the victims for their abuse or to frame poverty 
and homelessness as caused by laziness. For example, seeing communities of color as “diffi- 
cult,” “challenging,” or “risky” populations to study reflects deficit thinking among research- 
ers. For example, health communication scholars interested in obesity research among urban 
high school students might reinforce the individual deficit model by anchoring their research 
on low self-esteem, educational deficits, and poverty without taking into consideration fac- 
tors such as cultural identity, structural issues, and socialization. Or those studying mental 
health issues might emphasize medical treatment without examining how illnesses, healing, 
and treatment themselves might not be problematized the same ways in all communities, nor 
is access to health care a given for all. Often researchers interpret their results in ways that 
pathologize underserved communities and confirm existing stereotypes of the groups rather 
than challenging or confronting them. 

Since language is central to Communication scholarship, it is especially important to use 
culturally inclusive language in designing research materials. The language that is used in 
creating written and verbal communication such as questionnaires, materials to be used in an 
experiment, and consent forms should be sensitive to socio-political and cultural contexts. It 
is important to consider how research participants are characterized by research documents. 
For example, simple linguistic techniques, such as use of culturally appropriate pronouns (For 
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instance, members of the LGBTQ+ community instead of gay populations, depending on the 
context in which the research is being conducted), could make a huge difference in making 
research participants feel welcomed and included in the research process. 

While it is crucial to avoid potentially offensive expressions, it is equally important to 
ensure that the language used in research is mutually understandable. For instance, including 
a bilingual/multilingual survey option for populations where there is more than one language 
spoken is an important practice. Beyond the written documents, all interactions with research 
participants should be guided by principles of cultural responsiveness. When conducting 
research with groups that might differ from the researchers in terms of factors such as social 
class, race, ethnicity, religion, nationality, and education, it is vital to employ cultural sensitiv- 
ity and inclusive communication strategies. For example, while scheduling lab-based experi- 
ments in the evenings, it would be best to make sure that participants who need childcare 
have access to it or that the dates of the experiment do not coincide with popular religious 
holidays for minority groups. 

We can even see the role of quantitative research in Communication showing the ways in 
which researchers of color have an unequal opportunity to become published authors and be 
referenced by other authors in the Communication field. In a study cleverly titled “#Com- 
municationSoWhite,” Paula Chakravarrty, Rachel Guo, Victoria Grubbs, and Charlton 
Mcllwain (2018) use quantitative methods to determine that a disproportionate number of 
White authors have their studies published in Communication journals and have their studies 
referenced by other authors, as well. Specifically, the authors found: “In general, non-White 
scholars are underrepresented among the published first authors in communication journals, 
authoring only 746 of 5,262 (14%) documents published from 1990 to 2016” (p. 259) and that 
“articles authored by non-White scholars are cited significantly fewer times, on average, than 
White authors (1.7 to 2.5, respectively)” (p. 259). Not only do these data lay bare the impor- 
tance of inclusion in academic writing and publishing, they also demonstrate, once again, 
the power of numbers. Without the ability to identify these patterns across a wide swath of 
published studies—in the 5,262 articles from 12 major academic journals over a period of 
26 years that were examined in the study—possible only with quantitative research methods, 
the claims of bias and unequal access made by Chakravartty and colleagues would likely have 
been much less persuasive. 


Ethical Considerations Specific to Quantitative Communication 
Research 


Quantitative Communication research is empirical and data-driven by nature. Because quan- 
titative research typically involves collecting and storing large amounts of data, like hun- 
dreds of completed surveys, for instance, questions about where that data is housed, who 
has access to it, and what types of security measures are in place to assure confidentiality for 
those who provided the data are important considerations. Communication researchers, just 
like researchers in other fields, might have to report to IRBs that they will store data on a 
password-protected computer in a locked office that only they can access, for instance. When 
possible, researchers might also plan to remove any identifying information (names, places of 
residence, or any other details that might potentially reveal participants’ identities) from com- 
puter files containing the data. 

Increasingly, however, this issue is made more complicated in Communication and other 
fields that deal with “big data’”—including massive amounts of data often so large that tra- 
ditional computer processing is not sufficient to process that data. As Dove and colleagues 
(2016) point out, there is little guidance currently on data-intensive projects that are located in 
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multiple sites, especially across various nations. It becomes challenging when big data analyses 
involve international collaborative data-sharing where data are collected in one location but 
stored, linked, and accessed from other sites over long periods of time. It is crucial to have 
prior, ongoing, and sustained discussions with research institutions, ethics regulatory boards, 
and government bodies to create clear and transparent agreements of understanding. These 
discussions should include topics such as personal data breaches, monitoring and reporting 
of data use and storage, and procedural alignment across various governing and regulatory 
bodies. 

Increasingly, quantitative research is moving toward multi-site, multi-nation, multi-method 
approaches that involve collecting large-scale data, often including biometric and material 
data. 

Bioethics is an especially important consideration in quantitative Communication research, 
given the direction which several subfields have taken toward mobile health and online health 
initiatives, the collection of biophysiological data such as fMRI, skin conductance, heart 
rate, and even DNA and telemedicine. Some health communication scholars, for instance, 
are using DNA screening as data to interpret and track health conditions. The emergence 
of “empathetical technologies” means that mobile devices can sense and track media users’ 
moods and emotions just by the ways in which they physically and physiologically interact 
with technologies. While the power of mobile, sensory, and digital technologies has enabled 
better diagnosis, healing, and treatment, there is also potential for unintended harm to par- 
ticipants through violation of privacy and confidentiality, which needs to be taken seriously. 
Recently, revelations that law enforcement officials have used DNA databases intended for 
tracing genealogy for criminal investigation purposes raises important ethical questions for 
issues of privacy and confidentiality. 

Another emerging trend within quantitative Communication scholarship is the computa- 
tional analytics that include internet-based studies and the study of social media platforms 
for analyzing large-scale data. Currently, most Communication scholars using Twitter data 
are presenting conference papers and journal articles by assuming that such data are “public 
data.” However, it is questionable whether such data can be seen as anonymous and can be 
used for research without informed consent. An example of a study with an ethically murky 
design can be found in the case of Cornell University researchers working with Facebook to 
manipulate the news feed posts of over 600,000 people in 2012 to expose them to positive 
or negative messages in order to study emotional contagion without their explicit consent. 

A large percentage of tweets include personal information such as location, health infor- 
mation, etc. All research participants have the right to determine what information about 
themselves can be considered private and proprietary. Participants have the right to not share 
information, to withhold personal information, and to maintain secrets. In other words, 
researchers should respect the autonomy of participants and honor their preferences. 


Research in Action 


Applied ethics or situational ethics goes a step further to resolve specific moral issues 
using case studies, examples, and particular settings (Ellis, 2007). Here we share some 
case studies from Communication research for social justice. They are not only cases 
about protecting research participants, but they also consider ethics more broadly in 
terms of broader research practices within social sciences. These case studies have been 
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adapted from real-life experiences by the second author of the book (identities and 
locations have been changed). Case studies, like real life, might be ambiguous and not 
include all the information that you might want about the full scenario before you make 
a decision on how to proceed. As you analyze these case studies, feel free to make your 
own assumptions as you identify the key ethical issues to be resolved. To be clear, there 
are no right or wrong answers for these case studies. Use the ethical principles discussed 
in this chapter to analyze these cases. Some principles that could be helpful are non- 
violence, truth, justice, beneficence, respect, confidentiality, voluntary participation, 
informed consent, asset-based thinking, inclusion, ethics of care, the ethic of the heart, 
cultural responsiveness, relational ethics, bioethics, reflexivity, and privacy. 


Case 1: “You can either be a real scholar or a real activist, but 
not both.” 


You are an undergraduate health communication researcher in South Korea. You are 
presenting your research poster at a major conference. It is an analysis of sexual health 
conversations among LGBTQ+ teens of color. Another conference participant who 
stops by your poster says that you should reconsider your research area. He says that 
social science methods are diametrically opposite to critical social justice issues: “You 
can either be a real scholar or a real activist, but not both,” he says. He worries that every 
text you analyze is seen as sexist, racist, colonial, or classist and you are not defending 
your work with real data or logic. What are the ethical issues in this case study? How 
would you respond? 


Case 2: “Going virtual”: Conducting lab-based experiments in 
a crisis 


You are a Latina B.A. Communication student who is conducting research on gaming 
for your honors thesis at a Texas university. Your participants come to your lab to meas- 
ure skin conductance and heart rate as they play various types of games. In the middle of 
data collection, COVID-19 pandemic greatly affects Texas, including in the location of 
your research. What ethical issues do you have to consider as you decide whether or not 
to continue with the study? What is the role of legal, professional, and personal codes of 
ethics in making these decisions? Are there any conflicting values and principles? Who 
are the stakeholders to consult as you move forward in resolving the situation? 


Case 3: “Conducting race research is painful, tricky, and 
challenging.”’: A primer by an all-White research team 


A White male researcher from a European country writes to you, an African American 
researcher in the United States, about a primer on race research that he is writing up. 
He says, “We are a team of eight White scholars who are writing about how to conduct 
race research. We were wondering if you might be interested in reviewing this article 
and giving your feedback since you are an expert on race. If you are interested, we can 
also add you as a co-author. We are happy to share that the editor of a peer-reviewed 
International Communication Association journal has told us informally that they will 
be happy to publish this article. Their editorial term is ending in a couple of months. 
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So we are trying to get this piece to the journal as soon as we can before their tenure 
is over.’ When you open the research paper, the abstract says that “Conducting race 
research is painful, tricky, and challenging. We were never taught how to approach 
this topic while in graduate school. Therefore, we are writing this piece to help other 
scholars like us.” What are the ethical issues in this case study? How should you respond? 
What could the European researcher have done differently? 


Case 4: “Can I just sit in your group meeting?” 


You are an undergraduate Arab Muslim researcher in New Zealand. You conduct an 
online focus group discussion via Zoom about discrimination against Arabs and Mus- 
lims. At the beginning of the dialogue session, you announce: “This session is meant 
to be a safe space for Arab Muslim students to discuss Islamophobia. If you don’t self- 
identify this way, this might be a good time to leave this group.” Later on, in the small- 
group discussion, one of the participants says, “Actually I don’t identify as Muslim or 
Arab, but I just wanted to sit in your session. I am a White Christian digital religion 
scholar. I am writing a book on these topics.” What are the ethical issues in this case 
study? How would you respond if faced with this situation? What could you have done 
differently in designing this research study? 


Case 5: “I cannot afford to publish here.” 


You are a researcher in India who specializes in media literacy among youth. You receive 
an email from a professor in the Netherlands asking you to consider submitting your 
conference paper, which won a top paper award, to a journal that they edit. Since you 
are studying the role of gender in various media platforms such as TikTok and Snapchat 
in India, your research paper would be an excellent fit. You wish to browse through the 
recent issue of the journal. However, it is behind a paywall and your library does not 
have access to it. You write to the professor in the Netherlands requesting that they send 
you a copy of the current issue of the journal, which they do. As you browse through 
these articles, you notice that other researchers have conducted their work in media 
labs using expensive equipment or using nationally representative samples whose costs 
you cannot afford. You also notice a small sentence at the end of the message that says, 
“Since this is an open-access journal, the authors will have to pay a publishing fee of 
$1,000 if their paper gets accepted for publication.” What are the ethical issues in this 
case study? How would you respond if faced with this situation? What can be done to 
address such issues? 


Summary 


This chapter examines the core ethical principles that shape quantitative social justice research. 
The primary underlying ethical principle when working with research participants is to “do 
no harm.” What knowledge is created through research is just as important as how such knowl- 
edge is created, especially for social justice researchers. Ethics are often discussed mainly in 
terms of data collection in working with human participants. However, intrinsic ethical values 
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govern all aspects of research topics, questions, methods, participants, instrumentation, data 
collection, analysis, and dissemination. 

Egregious ethical violations such as those that happened during World War II, the Tuskegee 
Syphilis study, Milgram electric shock studies, and the Stanford prison experiments illustrate 
the history of such misconduct within social sciences. Ethical misconduct such as data fal- 
sification, fraud, oversight, and mistreatment of research participants can have severe conse- 
quences for both individual researchers as well as for the institutions of which they are a part. 
Institutional Review Boards (IRBs) were set up in response to the Belmont Report, which 
focuses on three main principles: respect, justice, and beneficence. Research participation 
should be voluntary and not coerced. Informed consent protects both researchers and par- 
ticipants. It includes information on what participants will be asked to do in a research study, 
the risks and benefits of participation, compensation, and procedures so that participants can 
make an informed decision about whether or not to consent to participating. 

Research ethics are influenced by procedural ethics, professional codes, and personal values. 
Procedural ethics focus on regulations and policies that govern research in an organization, 
state, or nation. Professional ethics are about codes set by professional organizations and net- 
works that a research belongs to. Personal ethics are about the individual researcher’s aspira- 
tional and individual moral principles. Especially for quantitative criticalists, it is important 
to be self-reflexive about one’s positionality and assumptions about power differences in con- 
ducting social justice research. Researchers should center ethics of care, be intentional about 
including underrepresented populations in research, and avoid tokenism and deficit thinking 
in research with minoritized groups. 

Within Communication, there are several interdisciplinary and international ethical considera- 
tions in various communicative contexts including ethical considerations in conducting internet 
research, social media, public data, bioethics, and computational analytics. The chapter concludes 
with several applied case studies that focus on situational ethics not just in terms of protecting 
research participants but also pertaining to the process as a whole. 


Discussion Questions 


1. What ethical dilemmas do you anticipate within research projects that you are working 
on or would like to work on? How might you plan to resolve them? 

2. How are procedural ethics different from personal (aspirational) ethics? What would your 
research project look like if you adhered to procedural versus personal (aspirational) ethics? 

3. Are there codes of ethics for professional organizations that you are a part of? What are 
your thoughts and opinions about these ethics principles? How would you modify them, 
if need be, and ensure that they are being followed? 

4. Reflecting on your own positionality as a researcher, what would it look like to be more 
culturally responsive, to be a humble listener, and to be more inclusive in your research 
approach? 

5. How do evolving technologies and media shape ethical considerations in research? 

6. What are some key ethical challenges to consider and overcome while conducting social 
justice research using new media technology and in online spaces? 
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3 Operationalization and Otherness 


Equipped with a sense of why a researcher may choose to select a quantitative methodology 
from the social science toolkit and how to use that methodology ethically as seen in Chap- 
ters 1 and 2, we are now ready to get into some of the nitty-gritty details about designing 
and carrying out such a project. There are many principles and practices that will help to 
ensure your research is both rigorous—that it is good science—and in line with social justice 
research—that it is good for society. Chief among those important considerations is decid- 
ing how you will define and measure the concepts you are interested in exploring in your 
research. 

We have called this chapter “Operationalization and Otherness” because operationali- 
zation, as you will read, is key to the process of arriving at ways to measure the topics and 
themes that interest you as a researcher. Otherness is a word used to signify how identities are 
shaped by social groups, with dominant groups (generally defined as those that have more 
economic and political power) typically set up in the center and non-dominant groups (those 
that have less economic and political power) often pushed to the margins. To “other” some- 
one is to make them feel excluded, left out, or otherwise diminished, which, in turn, protects 
and reinforces the power and privilege of those at the center. As researchers working toward 
social justice, we must do our best to be vigilant about whether decisions regarding how to 
measure concepts fall into the trap of othering people or failing to represent them and their 
opinions and experiences at all. 

The title of this chapter, then, is meant to suggest that the ways that we measure concepts 
have consequences, sometimes unintended ones. It is our responsibility to think through 
those consequences and be as responsive to human difference as possible when we are engaged 
in the measurement or operationalization stage of research design. We will return to this as we 
explain how operationalization is done in quantitative research and raise considerations and 
provide guidance about how to engage in the process as conscientiously as possible. 


An Introduction to Measurement 


Imagine that as a researcher you are interested in the ways that Latinx youth engage in “code 
switching’”’—changing how they converse based on the social context in which they are inter- 
acting. How will you know code switching has occurred in the young people’s conversational 
exchanges? To answer this question, you first need to define the concept, “code switching.” 
You need a careful, precise definition of what types of interactions or conversational turns you 
will decide fall both within and outside of its scope. Unless you have a clear definition of what 
counts as code switching and what does not, it would be hard to study it. 

Alternatively, perhaps you are interested in Communication education research, such as 
how undergraduates establish a deeper sense of their surrounding communities through civic 
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engagement and service learning. How will you decide whether that sense of the surrounding 
community has been achieved? What does it mean to have a deeper sense of the community? 
How will you know it when you see it? In addition to figuring this out, you will also have to 
come up with a plan for knowing whether Communication students engaged in civic engage- 
ment and service learning in the first place. In this example, then, as in most research studies, 
multiple concepts must be measured. 

Or maybe you would like to study how LGBTQ+ adolescents use social media to connect 
with other members of sexual minority groups and thereby explore and develop their own 
sexual identity. For this example, you will first have to determine how you will know whether 
LGBTQ+ adolescents are, indeed, connecting with others online and then decide how to 
measure whether they are developing their own sexual identity. You will also have to figure 
out how to measure use of social media as well as measuring LGBTQ+ identity. 

In order to answer research questions effectively, spending time upfront thinking about and 
clarifying how you will define and measure each of the key concepts in your study is crucial. 
Conceptualization is the process of defining terms used in a study, and operationalization 
is deciding how to measure or observe them. Social science methods expert Earl A. Babbie 
(2010) explains, “In social research, the process of coming to an agreement about what terms 
mean is conceptualization, and the result is called a concept” (p. 127). Concepts, then, can 
be a bit abstract and somewhat fuzzy, but they represent relatively shared and widely accepted 
notions of what terms mean. Operationalization moves our understanding of concepts from 
relative abstraction and ambiguity toward something much more specific and concrete. To 
operationalize concepts involves determining the specific steps the researcher will take to 
observe how concepts take shape in the social world (Figure 3.1). 

The task of measuring concepts is so important as to solicit this observation from psycholo- 
gist Kevin Cokley: “The integrity and trustworthiness of all research findings are only as good 
as the measurement devices used to generate the results” (2007, p. 226). Indeed, measure- 
ment is vital to quantitative research that is both scientifically and ethically sound. 


Progression of Measurement Steps 


Conceptualization 


Conceptual Definition 


Operational Definition 


Measurement in the Real World 


Figure 3.1 Measurement steps 


Source: Stephen Warren 
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Two Types of Definitions: Conceptual and Operational 


Two types of definitions are required when progressing from identifying the concepts that 
you would like to explore in your research to the ways in which you will measure those 
concepts. First, a conceptual (or theoretical) definition allows you to put into words the 
basic meaning of a concept. This is similar to a dictionary-type definition, and it represents 
the widely shared meaning of the concept. In developing a conceptual definition, you may 
want to draw from what prior researchers have come up with if your concept has been studied 
before, as long as you think those other researchers have done a reasonable job of capturing 
the meaning of the concept. 

If you are measuring hate speech encountered by individuals online, for instance, you likely 
have at least a hazy definition of hate speech in mind. Hate speech is the concept of interest. 
Yet, you also want to be sure that your conceptualization of the concept captures as much of 
the full range of meanings associated with the term as possible. You would be wise, therefore, 
to look up how others have defined hate speech in the scholarly literature. Then you can 
either apply a conceptual definition advanced by others or come up with one of our own if 
you find those lacking. In the particular example of hate speech, you might find some vari- 
ation in how others have defined it, since it is a difficult concept to define. Ben-David and 
Matamoros-Fernandez (2016), for instance, report that social media companies themselves 
have legal latitude to define and self-regulate hate speech on social media platforms. They 
have found that Facebook’s community standards prohibit “content that directly attacks peo- 
ple based on their: Race, Ethnicity, National origin, Religious affiliation, Sexual orientation, 
Sex, gender, or gender identity, or Serious disabilities or diseases” (Facebook, n.d., Section 2, 
para. 3). Perhaps that Facebook language is a good starting place for a conceptual definition 
of hate speech if you were to do a study in which you try to measure it. 

It is not enough, however, to develop and articulate conceptual or theoretical definitions 
for your concepts. You must also come up with operational definitions to plan for the 
exact and precise way you will measure each concept. Doing so will move your treatment of 
each concept from something somewhat vague and ambiguous to something concrete and 
identifiable. In the Ben-David and Matamoros-Fernandez (2016) study, they narrowed the 
concept of hate speech to a specific context (extreme right-wing Facebook users in Spain) 
and to particular expressions found online. Hate speech was operationalized as including 
negative targeting of blame toward groups such as immigrants as well as the use of slurs and 
insults related to these groups. From the broad concept of hate speech, therefore, Ben-David 
and Matamoros-Fernandez use more specific and measurable observations to operationalize 
the concept. 


Operationalizing Indicators and Dimensions of Concepts 


When concepts are broad or rather ambiguous, one useful approach in their measurement is 
to think of aspects of the concept you can observe that would indicate the presence or absence 
of the concept. Rather than try to get at all of the many meanings and manifestations of a 
broad concept in a single measurement, for instance, the social science researcher can deter- 
mine aspects or manifestations to observe that would indicate that the concept is operating in 
the social world. Particular observations that show the presence or absence of a concept are 
called indicators. 

To add even more richness to ways to measure a concept, we might consider dimensions 
of the concept, as well. Dimensions are aspects of concepts that group together logically, 
sort of like subgroups operating within the overall meaning of the concept. The concept of 
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aggression might have dimensions that include physical aggression and verbal aggression, for 
instance. The concept of prejudice might be broken down into affective (involving emotions 
and attitudes) and cognitive (involving thoughts and conceptualizations) components. The 
same concept, therefore, can have components that are distinct from each other but still tap 
into the same, shared overall concept. 

Complex concepts may take on many forms, and therefore the measures you use to observe 
those concepts may well reflect multiple dimensions and/or might rely on the use of multiple 
indicators. For instance, in one of the first author’s prior studies, she was interested in study- 
ing “hypermasculinity” in characters that appeared in magazine ads for video games (Schar- 
rer, 2004). Based on prior research and theorizing, she began with a conceptual definition 
of hypermasculinity as being tough, callous, aggressive, and physically imposing. She looked 
closely at an operationalization that Mosher and Sirkin (1984) had created to measure hyper- 
masculinity as a personality type in individuals. She adapted the Mosher and Sirkin opera- 
tionalization of the personality traits of hypermasculinity to the ways that characters appearing 
in the ads for video games were depicted. To do so, rather than just asking, “Is this character 
hypermasculine?”—which would call for a very broad application of a complex concept— 
she looked at indicators of hypermasculinity. Those indicators included depictions of the 
characters using or holding weapons, making threats, or being very muscular. Given the 
conceptual definition of hypermasculinity, there would likely be widespread agreement that 
any one of those particular depictions could be understood as indicators of the concept. In 
fact, prior researchers (Hall, 1992; Mosher & Sirkin, 1984) had identified four components of 
hypermasculinity in their measures of the concept: having callous attitudes, especially about 
sex; perceiving violence as “manly;” being drawn to danger and thrill; and suppressing emo- 
tions that might be perceived as weak. These components of the concept demonstrate its 
dimensions, and in the study of video game ads, Scharrer made sure to measure indicators 
of each dimension. 


Variables, Values, and Attributes 


In order to delve into the topic of measurement, we must introduce the definition of a vari- 
able. A variable is any concept of interest in a study that varies, taking on more than one 
value. The opposite of a variable in quantitative social science is a constant, something that 
does not vary. For instance, if you are interested in exploring views of news coverage among 
Muslims living in the United States, Muslim identity is not a variable in your study; it is a con- 
stant. All of your research participants identify as Muslim. Or if you show a group of research 
participants a public service announcement (PSA) about organ donation and then seek to 
measure whether they are persuaded to become donors themselves, exposure to the PSA is a 
constant since all participants experienced it in the research design. 

Unlike constants, variables take on multiple attributes, the range of qualities or charac- 
teristics that constitute a variable. For instance, imagine that you are studying whether high 
school-aged students have different opinions about the March for Our Lives protest of 2018 
(a student-organized protest against gun violence in the United States spurred in large part 
by school shootings) compared to older research participants. You could use “high school- 
aged students” and “those older than high school-aged students” as the attributes of a variable 
to classify research participants into each group. Or, if you are conducting an analysis of the 
depiction of women starring characters in action films, gender of the starring characters is 
an important variable. That variable can take on multiple attributes, including woman (a cat- 
egory that can include cisgender—those whose biological sex matches their gender identity— 
and transgender— women those whose biological sex does not match their gender identity), 
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man (again, including cisgender and transgender), non-binary, or gender fluid. Once you 
have established how to determine the character’s gender, you will also need a mechanism for 
determining whether the character is in a starring role. The starring role variable might have 
two attributes: yes or no. 

When you study opinions using survey methodology, to operationalize variables you have 
to decide both how to phrase the survey item itself and what options you will use for respond- 
ents’ answers. The operationalization is the combination of the two (the survey item and the 
response options for the survey item). For instance, gathering data about charitable giving 
during the quarantine following the COVID-19 pandemic, you might include “It is impor- 
tant for me to provide financial support to the local food pantry to help those in need during 
the pandemic” as your survey item and have response options of “strongly agree,” “agree,” 
“neutral,” “disagree,” and “strongly disagree.” In this example, respondents have five differ- 
ent options to use to indicate their opinion. Each of those five options is an attribute of the 
variable measuring the degree to which respondents found supporting the food pantry to be 
important. 

The final step in operationalizing variables is to transform attributes into values, which 
are the numerical properties assigned to each variable for the purpose of statistical analysis. In 
quantitative research, ultimately, as we established in Chapter 1, the data generated must be 
represented by numbers rather than words and phrases. Opinions on surveys may be expressed 
as scores measuring agreement on a scale of 1 (strongly disagree) to 5 (strongly agree), for 
instance. Or a numerical value may be generated by numbering options on a continuum, 
asking, for instance, how much of an introvert or an extrovert research participants consider 
themselves to be, using values that include: 1 = very introverted, 2 = introverted, 3 = neither 
introverted nor extroverted, 4 = extroverted, and 5 = very extroverted. Or, research partici- 
pants might report, for example, an estimate of the number of minutes per day that they spend 
on social media. Features of texts may be quantified by counting up the number of instances 
in a set of media content that share a common trait, like the number of acts of aggression in 
trailers for films with top box office receipts or the number of Asian characters that appear 
in recurring roles in Netflix original programs. Features of texts might also be represented 
numerically by noting the presence or absence of some attribute—where, for instance, a 1 is 
used to indicate that a recorded face-to-face conversation includes some form of conflict and 
a 0 indicates that it does not. Regardless of whether people or texts are being studied, the 
data that is gathered and analyzed by quantitative researchers is always expressed numerically. 

Representing data numerically has its pros and cons, in general, as well as in terms of its 
implications for advancing the goals of social justice. Numbers have a fairly cut and dried, 
straightforward, widely accepted meaning. So when a person on a survey reports, for instance, 
watching TV for an average of three hours per day, we all know what three hours per day 
means. We can contrast this way of knowing about television exposure with an approach that 
uses words and phrases to collect information. If a respondent says they watch “a lot” of tel- 
evision or “not much at all,’ we might have varying interpretations of what those words and 
phrases mean, which introduces some ambiguity. Numbers, then, are precise and have widely 
shared meaning, which allows for a fairly consistent interpretation. 

On the other hand, that exact same precision also has its downsides. In being precise, using 
numbers as a means of collecting information about people and texts can also be narrow and 
confining, not sufficiently responsive to richness of detail and complexity. If a researcher is 
interested in studying people’s opinions about a particular issue, for example, they may use 
a closed-ended measure with a predetermined and finite list of possible responses like, 
“T think climate change is one of the most pressing concerns in society today” with a range of 
choices provided, such as 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 
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Figure 3.2 A cartoon about quantification 


Source: Serg, http://lightbulbcartoon.com/Research.html 


4 = agree, and 5 = strongly agree. Or they may use an open-ended measure like, “What 
is your opinion about the importance of climate change as a contemporary issue? Please 
explain” and provide respondents with the space to write in a detailed answer. Provided that 
respondents do, indeed, take the time and care necessary to respond, we would likely know 
more about the nuance, fullness, and depth of their views in this open-ended way of posing 
the question compared to the closed-ended measure. 

For social justice research, researchers must be careful to ask questions or pose statements 
and provide a variety of potential responses that do the best job possible of capturing as much 
of participants’ views, experiences, and responses to the question as possible. The concept of 
voice is a crucial element of social justice research—who gets to speak, who will be listened 
to, and whose input matters. Therefore, making sure the options available to capture the voice 
of research participants are imbued with as much meaning and importance as possible is a key 
way forward for the critically conscious quantitative researcher. If the study is with people, 
like a survey or an experiment (more on those forms in subsequent chapters), always remem- 
bering that the numbers in the data represent real people whose voice we have a responsibility 
to represent in our studies is vitally important. No one wants to feel like a number, and no 
one should have to do so. 


Using Closed-Ended Measures 


Because quantitative research uses data represented by numbers, to measure variables it typi- 
cally employs closed-ended measures with a predetermined selection of values rather than 
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open-ended measures in which responses or observations are filled in using words. This puts 
the onus on the researcher to anticipate any possible response and come up with a full range 
of options to cover all responses. 

Imagine that you have asked research participants to read a news article on the controversy 
surrounding the U.S. Supreme Court nomination of Brett Kavanaugh, who was accused of 
sexual assault by Dr. Christine Blasey Ford in 2018. You want to know whether they think 
the news article is biased. You can try to find that out using an open-ended item such as: After 
reading the news article provided, please use the space below to explain whether you think the news story 
is fair or whether you think it is biased in some way. 

Or you can try to find out whether participants think the news article is biased using a 
closed-ended item: After reading the news article provided, please consider each of the following state- 
ments, circling the response that best reflects your view of the news story you read. 


This news story is fair. 

strongly agree agree neither agree nor disagree disagree __ strongly disagree 
This news story is biased. 

strongly agree agree neither agree nor disagree disagree strongly disagree 


Quantitative research usually uses closed-ended items because they are easier to translate into 
numerical values. The choice for the research participant of one fixed response from a set of 
closed-ended items is less open to interpretation by the researcher compared to the inter- 
pretation needed to handle open-ended responses. For our example here, for instance, the 
“strongly agree” can be a 1, “agree” a 2, etc., and the assignment of those values is straight- 
forward and requires very little researcher interpretation. Any researcher will look at the 
closed-ended data in the same way (e.g., assigning a “strongly agree” choice with a 1), which 
strengthens claims of objectivity. As we discussed in Chapter 1, research that strives to be 
objective can sometimes be persuasive to people in policy making or other decision making 
roles, which can, in turn, bring about change. 

But it is possible to quantify answers to the open-ended item, too, actually, and there are 
some potential advantages to using open-ended items for some research topics or the meas- 
urement of some concepts. Open-ended items can make the possibility of othering people 
(making them feel outside of the norm) a moot point since respondents choose how to rep- 
resent themselves rather than having to answer within the response options determined by the 
researcher. They have the benefit of allowing the person to describe something using their 
own terms and in their own words. 

However, on the downside, for quantitative research, the responses to open-ended meas- 
ures such as these incur an additional step, coding those responses, which means interpreting 
the responses and imposing a way to categorize them and then assigning numbers to those 
categories. Not only does this take additional time, but this step also introduces the potential 
for the researcher to misinterpret or misrepresent the responses, as it requires the researcher’s 
judgment about how to code the responses. Coding open-ended responses into numbers 
also inevitably reduces some of the complexity that open-ended measures invite. Using the 
example of the news story on the Supreme Court nominee, a quantitative researcher using the 
open-ended measure would summarize a respondent’s thoughts by grouping those thoughts 
together with similar thoughts expressed by others in the sample and then assigning that group 
a number in common. For instance, the researcher may classify all of the respondents who said 
the news story was biased but did not give a reason with a 1 and all of the respondents who 
said the news story was biased and did provide a reason a 2. Finally, open-ended measures 
are also more time intensive for research participants who have to fill them in, and given that 
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they do take time, research participants may be tempted to sidestep these items or shorten 
their answers. 

There are two basic requirements when operationalizing variables in a closed-ended 
manner. The values provided in response to the item must be mutually exclusive, 
meaning that the values for the variable should not overlap. For most variables, it is ideal 
to have only one response that represents the best possible option, so that, for instance, 
survey respondents are not registering their opinion as both “agree” and “disagree” if they 
really mean “neither agree nor disagree” or that they feel “neutral.” And the values must 
be exhaustive, so that there is a mechanism to record every possible response. There 
should never be a case where there is no way to record what the survey respondent would 
like to say about the issue. Every possible value that the variable may take on should be 
anticipated. 

Imagine that an organizational communication scholar would like to ask survey respond- 
ents to estimate the number of minutes they spent reading the organization’s sexual harass- 
ment policy. She uses the following operationalization to do so: 


About how many minutes would you say you spent reading your company’s sexual harassment 
policy? 


1—5 minutes 

5—10 minutes 

10-15 minutes 
15—20 minutes 

More than 20 minutes 


These response options might look OK at first glance but this example actually violates both 
of the principles of variable construction. The response options are not mutually exclusive. 
Rather, they overlap. If a respondent wanted to indicate that they spent 5 minutes on the 
policy, for instance, there are two options they could select rather than just one, given that 
the first and second option both have 5 minutes listed. The same problem would occur for 
respondents who want to report that they spent 10 or 15 minutes. Also, the response options 
are not exhaustive. The scholar wisely used “More than 20 minutes” to cover any number of 
minutes beyond 20 that was spent on the policy. But what if the respondent did not read the 
policy at all? The scholar should add a “0 minutes” or “I did not read the policy” option in 
order to exhaust all possible responses. 
The corrected operationalization, therefore, would look like this: 


About how many minutes would you say you spent reading your company’s sexual harassment 
policy? 


I didn’t read the policy. 
1—5 minutes 

6—10 minutes 

11—15 minutes 
16—20 minutes 

More than 20 minutes 


Now the response options are both mutually exclusive and exhaustive. 
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On occasion, there is an exception to the general rules of variable construction regarding 
the mutual exclusivity “requirement.” You can factor in the possibility that two or more values 
rather than just one might be at play in a single operationalized variable. For instance, in a 
survey item, you can invite respondents to “Check all that apply” in a list of response options 
or to “Rank these five reasons for supporting renewable energy from 1 for least important to 
5 for most important.” In using techniques such as these, you do not have to force respond- 
ents into just one response and, in some cases, you can better account for the complexity of 
experience and opinion. For the most part, however, the principle is that one and only one 
option should be selected for each variable. 


Levels of Measurement 


Once you begin experimenting with the operationalization of concepts—deciding how, 
exactly, you will measure your variables—a key consideration is deciding what level of meas- 
urement to use. Researchers have identified four different “levels” that variables can take on. 
An understanding of levels of measurement is one of the most important building blocks in 
learning quantitative research methods, so please read and consider this section of the chapter 
carefully. Commit to practicing and internalizing the logic of levels of measurement, and 
when it comes time later to learn the principles and techniques of statistical analysis, you will 
be glad you know how to determine levels of measurement. 


Categorical Variables 


A categorical variable is any variable that uses categories for values. In categorical variables, 
values are grouped together. Here is an example: About how many texts would you say you send 
each day, on average? 0 texts, 1-10, 11-20, 21—30, 31—40, more than 40. For this way of meas- 
uring the frequency of sending texts, those who would have answered anywhere between 1 
and 10 are categorized together as are those who would have answered any number between 
11 and 20, etc. There are two levels of measurement for categorical variables, nominal and 
ordinal level variables. 


Nominal Variables 


A nominal level of measurement is used when the values of the variable are arbitrarily 
assigned to numbers in order to classify responses based on a shared characteristic. The num- 
bers assigned, however, are only used “nominally” (thus the name!) and do not actually signify 
more or less of something. For example, if you have participants in an experiment, you might 
use the label 1 to classify those in the control group and 2 to classify those in the treatment 
group, but you could easily reverse those labels and that would be equally valid. The control 
group does not have to use the label of 1. Giving them a 1 is an arbitrary decision. Here is 
another example. If you adopt the Pew Research Center’s response options for measuring 
religious identity, you could use the following labels to classify individuals: 1 = Christian, 
2 = Muslim, 3 = Hindu, 4 = Buddhist, 5 = Jewish, 6 = Unaffiliated, 7 = folk religions, 
8 = other religions. But, once again, you could mix up the order of the numbers, too, and 
nothing meaningful would be changed. The numbers assigned to nominal level variables are 
arbitrary and they do not represent some mathematical quantity. Nominal measures are the 
lowest form of measurement because they are the most basic, but as we will see they can be 
quite handy in Communication research. 
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There is also a particular type of nominal level variable called a dichotomous variable. 
A variable that has just two values is called a dichotomous variable. For example: Have you 
ever been cyberbullied? Yes or No has just two options and therefore can be called dichotomous. 


Ordinal Variables 


An ordinal level of measurement does not use arbitrarily assigned values but rather uses val- 
ues that ascend or descend, with changes in values in this form of measurement representing 
greater or lesser quantity of whatever is being measured. In other words, there is an order or 
a ranking to the values used for an ordinal level variable. In media research, for instance, we 
might ask about the frequency with which survey respondents spend time with a particular 
form of media, like YouTube or cop shows or public affairs programming. We might ask: 
“How frequently do you watch YouTube videos? Please select the option that most closely 
matches your response” with response options that include, “Several times a day or more,” 
“About once a day,’ “Once or twice a week,” “About once a month,” and “Less than once a 
month.’ In this case, the response options have a logical order, beginning with the most fre- 
quent and ending with the least frequent response, with each response in between following 
the logic of descending frequency. Ordinal level variables, therefore, both classify responses 
into groups (like nominal) and order the values of the variable, thereby providing a useful way 
of remembering how to recognize this type. 


Continuous Variables 


The opposite of a categorical variable is a continuous variable, which is also known as 
(using the term in Figure 3.3) a metric variable. A continuous or metric variable allows 
specific and precise numerical values to be recorded rather than utilizing groups of values or 
categories. About how many texts would you say you send each day, on average? ___ texts is an 
example of a continuous measure. Respondents can write in any number from 0 to goodness 
knows how many! Ultimately, their answers could be arrayed on a continuum from lowest to 
highest (thus the name continuous). There are two levels of measurement for continuous vari- 
ables: interval and ratio. 


Interval Variables 


For the interval level of measurement, the logic used in determining the values is similar 
to an ordinal level variable in that response options can be ranked in ascending or descend- 
ing order, and just like both nominal and ordinal variables, an interval level variable classifies 
values. Yet, in an interval level of measurement, unlike in ordinal, the distance between each 
response option is known, fixed, and assumed to be equal. In other words, the values of the 
variable are placed at equal intervals, thus the name. However, by definition, an interval level 
variable also does not have a meaningful value of zero that would signify the complete absence 
of the variable. A frequently cited example is IQ (see Babbie, 2016, for instance). You could 
say the difference between an IQ of 90 and 100 is the same as the difference between an IQ 
score of 75 and 85 (each varying by ten-point intervals), but an IQ of 0 is not possible since 
everyone will have some sort of score on the IQ. 

In Communication, researchers treat Likert scale variables as measured at the interval 
level (Keyton, 2015). The Likert scale is the commonly used survey research technique in 
which a statement is provided to respondents, and they are asked to provide their level of 
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endorsement of that statement with response options such as “completely disagree,” “some- 
what disagree,” indicate that they “neither agree nor disagree,” “somewhat agree,” or “com- 
pletely agree” (Figure 3.4). In Communication research (as in many other fields), we assume 
the distance between, say, “completely agree” and “somewhat agree” is the same as the dis- 
tance between “somewhat agree” and “neither agree nor disagree,’ etc., thereby meeting the 
requirements of the interval level of measurement. You can change the number of response 
options in a Likert scale from five to seven or more, but many analysts contend that you 
should use an odd number so that there is a neutral point in the middle with equal numbers 
of options on either end. You should also keep in mind that if you use too many options, 
it may be difficult for those answering to parse their sentiments into such fine distinctions 
(like, deciding whether to select a 5 or a 6 on a 7-point scale if they are moderately in favor 
of the statement). 


Ratio Variables 


Finally, the most precise level of measurement is the ratio level. Ratio level variables share 
the qualities of the preceding levels of measurement in that they classify, order, and have equal 
intervals. For ratio level variables, however, unlike for interval level variables, a value of 0 
does, indeed, mean the absence of this variable. For example, you could count up the number 
of U.S. mainstream news stories that use the words terrorism and Muslim together and compare 
that number across different sources. Perhaps news channel one had zero such simultaneous 
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references, but news channel two had 10 and news channel three had 20. In this case, the zero 
is meaningful. It means there were no times in the news channel one content in which the 
terms Muslims and terrorism were used together. And you can say that news channel two had 
half as many such references as news channel three. 

One particular example of a ratio level measure format used fairly commonly in public 
opinion research is the feeling thermometer. A feeling thermometer asks respondents to 
indicate how warmly (favorably) or coldly (unfavorably) they feel about someone or some- 
thing, using a physical representation of a thermometer. Each respondent can select a precise 
value between 0 (for the coldest possible feelings) and 100 (for the warmest possible), and 
therefore there is considerable potential for variation in the exact selections expressed by a 
sample of respondents. As you can see in Figure 3.5, the Pew Research Center used a feeling 
thermometer to compare how Republicans and Democrats in the United States feel about 
each other as well as about other groups like military personnel, atheists, and elected officials 
in Washington, D.C. We can interpret the data in the figure to indicate that although those 
in both major political parties in the U.S. have favorable feelings toward military personnel, 
feelings diverge by political party affiliation for other groups. 

In practice, the difference between interval and ratio level variables has been somewhat 
difficult to determine and is also rather a moot point, since as we will see in Chapters 8 and 
9, many common statistical tests treat interval and ratio level variables interchangeably. There- 
fore, some prefer considering interval and ratio level variables together and calling them either 
continuous or metric variables. 


Deciding What Level of Measurement to Use 


Many times there are multiple ways to operationalize a variable that cut across more than one 
level of measurement (as you can see with the frequency of sending texts examples we used 
earlier). Researchers may have a decision to make, therefore, about what level of measurement 
to use (Figure 3.3). 
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Figure 3.5 Feeling thermometer 


Source: “Partisanship and Political Animosity in 2016.” Pew Research Center, Washington, D.C. (June 22, 2016) www. 
pewresearch.org/politics/2016/06/22/partisanship-and-political-animosity-in-2016/ 


To measure age, for instance, you can ask research participants: “How old are you?” and pro- 
vide a blank with a label giving the metric with which you would like participants to answer, 
like this: “ years old.” (Its important to provide the metric rather than just a blank line to 
fill in—in this case, specifying that you want folks to answer in years—so that, for instance, 
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some smart aleck refrains from writing “Really old!”’). This is a ratio level variable. Each 
participant will put in a specific age in years, and the ages of all of the participants could be 
displayed on a continuum from youngest (the minimum value for age) to oldest (the maxi- 
mum value for age). Registering an age of 0, although quite rare in practice, is a meaningful 
possibility, since infants younger than one year of age would have this value for age (of course, 
we would have to assume someone else is answering the survey item on their behalf!). Con- 
versely, you could use the age groups that the Nielsen company uses most often to measure 
television viewing and ask, “How old are you?” and provide categories of responses, such as 
“18-34,” “35—49; and “50+.” This would place all research participants into just these three 
groups (assuming you are not studying those under the age of 18) and would be considered an 
ordinal level of measurement since the responses ascend but at unequal intervals. 

In general, when you are faced with multiple ways to measure the same variable and you 
only want to or can use one, it is usually best to use a higher level of measurement like interval 
or ratio for three reasons. First, higher levels of measurement provide the most information. 
You know more about your research participants, for example, if you know the full range of 
their exact ages than if you categorize them into three age groups. In research, it is usually 
best to gather more precise and detailed information when you can. Second, you can always 
go from more precision to less precision by later categorizing a variable that was measured 
as a continuous level variable, but you cannot do the reverse. You cannot make a categorical 
variable continuous. In other words, if you do ask for precise ages but later you want to see 
whether various age groups differ from each other, you can use that ratio level age variable 
to place each participant into age groups. But if you only had the three age groups to begin 
with, you have no way of knowing exact ages. 

Finally, as you will see in Chapter 9, higher level variables allow for more complex statisti- 
cal procedures in data analysis compared to lower level variables. This is primarily because 
lower-level variables do not allow for the calculation of an average. For nominal level data, for 
instance, since the numerical values of the response options are arbitrarily decided and have 
no mathematical value, an average does not make sense. Somewhat similarly, for ordinal level 
data, since the distance between each value is not equal, calculating an average is likely to be 
unwise, as well. If you measured frequency of checking social media accounts as “almost con- 
stantly,’ “many times per day,’ “a few times a week,” and “less than a few times a week,” the 
interpretation of the average would be very fuzzy, indeed. Metric variables have meaningful 
averages and allow for the largest number of mathematical functions. 
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Validity 


In this book, we refer often to “social justice research,” and in Chapter 1 we provided a basic 
definition of what that entails. By advancing Frey and Carragee’s (2007) assertion that social 
justice research involves challenging oppression, social inequalities, and power imbalances faced 
by marginalized groups, we provided a conceptual definition of social justice research, giving 
some sense of what we mean by the term. Yet, social justice is an example of a concept that is 
complex, broad, and multifaceted, and there might be many different understandings of the term. 

If you could somehow visualize and map out every person’s entire understanding of a 
complex concept such as social justice, perhaps that visualization would take the shape of an 
enormous cloud with hazy and ambiguous edges, somewhat similar to the word cloud image 
that we have included here (Figure 3.7). No single measure used in social science research 
will get at all of that complexity or will reproduce the whole cloud. However, the duty of the 
researcher is to try to get at as much of the content of the cloud as they can. When readers 
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Research in Action 


The Operationalization of Race and Ethnicity in the U.S. Census 
The U.S. Census uses five categories for respondents to report their race: 


American Indian/Alaska Native 

Asian 

Native Hawaiian or Other Pacific Islander 
Black or African-American 

White 


and two categories for respondents to report their ethnicity: 


Hispanic/Latina(o) 
non-Hispanic/Latina(o) 


Since 1997, the federal definition allows respondents to select more than one 
category for these. 

* Source: U.S. Office of Management and Budget, Directive 15 Race and Ethnic Standards 
for Federal Statistics and Administrative Reporting, https: //transition.fcc.gov/Bureaus/OSEC/ 
library /legislative_histories/1195.pdf 


Here’s a table showing the percentage of individuals who characterized their race and 
ethnicity using the U.S. Census categories the same way in both the 2000 and 2010 
censuses. 


Non-Hispanic Whites 97% 
Non-Hispanic Blacks 94% 
Non-Hispanic Asians 91% 
American Indians or Alaska Natives 53% 
Native Hawaiians or Pacific Islanders 48% 
Hispanics 41% 
Non-Hispanic Multiracials 23% 


*Source: Liebler, Carolyn A., Sonya Rastogi, Leticia E. Fernandez, James M. Noon, and 
Sharon R. Ennis. 2014. America’s churning races: Race and ethnic response changes between 
Census 2000 and the 2010 census. Revision of Center for Administrative Records Research and 
Applications (Census Bureau) Working Paper 2014-09, Washington, D.C. 


Questions to consider: 


How well do you think the U.S. Census operationalization captures the concepts 
of race and ethnicity? Why do you think members of some racial or ethnic groups 
varied in how they reported their race or ethnicity? Do you think it matters that 
there is this variation in how U.S. residents report their race and ethnicity? Why 
or why not? 
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Figure 3.7 Social justice word cloud 


Source: Stephen Warren made using Wordclouds.com 


of a study learn of how the researcher conceptualized and operationalized the concept in the 
study, those decisions should make sense to the readers. Readers should find at least some 
connection—albeit potentially imperfect—between their own understanding of the concept 
and the ways the concept was treated in the research. 

This is the concept of validity, which is traditionally defined in social science research as 
whether researchers are measuring what they purport to be measuring, whether their meas- 
ures are doing a reasonable job of tapping the meaning of the concepts at play. It is helpful 
to think of validity as closely related to accuracy. Are the measures used to capture a concept 
zeroing in on that concept’s meaning accurately? There are a number of types of validity, and 
researchers can use those types as guidelines in research design to consider the validity of their 
measures. 

First, face validity is the degree to which the measurement maps on to a common under- 
standing of the concept. Does the operationalization appear to make logical sense by reflect- 
ing at least some of the shared meaning attached to the concept it measures? If so, it is valid 
“on its face” and has adequate face validity. This is the easiest form of validity to achieve, since 
it simply has to do with whether measures appear logically to tap into the concept of interest. 

Second, you can determine whether the measurement relates in a logical manner with 
another variable outside of your study. This is called criterion-related validity. Communi- 
cation researcher Barbara A. Ritter (2014), for example, came up with new measures to study 
perceptions of the online environment and sexual harassment in cyber settings. Because they 
were new, she took steps to determine whether those measures were valid. She tested crite- 
rion-related validity by comparing one of the new measures she had created to a previously 
existing measure designed to tap a similar phenomenon. If the new measure Ritter created 
was valid, it should be correlated with the already existing measure, and it was. 
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Finally, content validity refers to the degree to which the full range of meanings of the 
concept are being reflected in the measurement and how the components all relate to each 
other. If researchers say they are measuring the degree to which female action heroes can be 
considered feminist role models, for instance, they should consider multiple ways in which 
“feminist role models” might take shape, and those various components should relate to one 
another. For instance, the degree to which female action heroes rescue others should relate to 
whether they display self-reliance if both are good indicators of being feminist role models. 


Reliability 


In addition to validity, an equally important element of research design is the reliability 
of measurement, defined as the degree to which the same measurement procedure would 
yield the same result if put into place multiple times. Does the measure lead to generally 
quite stable and consistent patterns? If so, it is reliable. Or does the same measure produce 
results that vary wildly across different contexts? That is the sign of an unreliable measure. 
If you ask the same research participants the degree to which they support equal pay for 
equal work regardless of the worker’s gender at two different points in time, for example, 
you will hope that their responses would be more or less the same both times. If they 
are, we would consider our means of generating this data—the way we have asked the 
question—to be reliable. 

In their book Quantitative Research Methods for Communication, James S. Wrench, Candace 
Thomas-Maddox, Virginia Peck Richmond, and James C. McCroskey (2008) note, “If we 
measure something on several different occasions and never get anywhere near the same score, 
there are only two conclusions we can draw: either the thing we are measuring is constantly 
changing or our measuring instrument is worthless” (pp. 201-202). The second possibility is 
related to reliability, and Wrench and colleagues point to three factors that can contribute to 
issues with reliability. 


¢ First, the way that items are created can be ambiguous or unclear. Generally, we 
can expect more reliable data when measures are clear and precise. Contrast asking 
research participants the degree to which they support “equal pay for equal work 
regardless of the worker’s gender” with their support for “gender and labor justice.” The 
former is fairly specific, whereas the latter is broader and more open to interpretation. 
We would expect answers to the former to vary less within participants across multiple 
iterations of being asked compared to answers to the latter, given the specificity and 
clarity of the phrasing. 

* Second, Wrench and colleagues (2008) suggest that the length of the research instru- 
ment can impact reliability. Generally, multiple items rather than single items used to 
measure a concept will produce more reliable results. This is because any single item can 
produce “error” in that it may be unclear or confusing to research participants. So, using 
more than one item helps protect against such errors highly impacting the data. But, of 
course, you do not want your study to take too much of your research participants’ time. 
If they are burned out, bored, or frustrated, that can produce unreliable results, as well. 

e Third, the manner in which the research is administered is important. Providing 
research participants with clear instructions or guidelines is vitally important, for instance, 
as is establishing a professional setting for data collection. 


Just like with validity, there are a number of ways to formally test reliability. You could employ the 
test-retest method, which simply means asking the same question (or set of questions) twice to 
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see the degree to which responses vary, as we have illustrated with our equal pay for equal work 
example. Or, if you have a number of indicators of your concept, you could use the split-half 
method to verify whether any randomly selected half of the items used to measure a concept will 
produce the same results (more or less) as the other half. Pablo Davanzo and colleagues (2004), for 
instance, undertook a Spanish translation of the Child Depression Inventory and tested its reliabil- 
ity when administering it to a sample of Spanish-speaking youth. The items on the questionnaire 
were numbered from 1 to 27. To conduct a split-half test of reliability, Davanzo and colleagues 
compared the children’s responses to the items with an odd number to their responses to the 
items with an even number. Because they found a high correlation between the two groups of 
scores, their claims that the Spanish translation was reliable were strengthened. In other words, the 
children in the sample responded similarly to the odd-numbered items and the even-numbered 
items, suggesting that both sets of items were producing reliable results. 

If you are categorizing responses to open-ended questions as we discussed earlier in this 
chapter, comparing your judgments to the judgments of others can provide a check and bal- 
ance on the potential for your own subjectivities to threaten reliability. In fact, in our chapter 
on content analysis (Chapter 7), we will learn about intercoder reliability, a means of com- 
paring observations about texts across different observers. For example, if your study calls for 
determining how much objectification of women occurs in music videos, you will want to 
show that the research design you have developed for studying that question produces more 
or less the same results no matter who is actually making the observations. And finally, if you 
are using multiple items to measure the same concept, you can run reliability statistics like 
Cronbach’s alpha to determine whether the items do, indeed, have what we call “internal 
consistency,” meaning whether they relate well together as a group. We will explain those 
statistics more in Chapter 9. For now, all that is important to keep in mind is that each of 
these checks on reliability can tell researchers how well they have achieved the goal of using 
measures that produce consistent, trustworthy, and dependable results. 


Balancing Validity and Reliability 


One choice researchers often have to make is whether they will use existing measures that 
other scholars have developed and put into practice already or whether they will try their 
hand at coming up with their own. Each option has potential advantages and disadvantages. 
You might find that using existing measures is more reliable since those measures have been 
tested already and perhaps have already been shown to work well. Yet, designing your own 
measures might be more valid if existing measures only partially get at the phenomenon you 
are trying to explore. If you design your own measures, you can tailor them specifically to the 
concepts and topics you are studying, and therefore they may have greater claims on validity 
compared to those that were generated by others for a different purpose. 

The question of whether to use existing measures or design them yourself relates to a 
broader discussion of the ways in which validity and reliability are balanced in research design. 
In fact, oftentimes in practice, validity and reliability are at odds with each other. When we 
attempt to make a measure simple, precise, specific, and straightforward to bolster reliability, 
we are simultaneously threatening its validity as we run the risk of oversumplifying a complex 
concept. Earl Babbie, the author of one of the most popular textbooks on research methods, 
The Basics of Social Research (2016), describes the tension between reliability and validity as a 
“persistent and inevitable dilemma,’ by pointing out that: 


Most of the really interesting concepts we want to study have many subtle nuances, and 
it’s hard to specify precisely what we mean by them. . . . Very often, then, specifying 


Operationalization and Otherness 65 


reliable operational definitions and measurements seems to rob concepts of their rich- 
ness of meaning. . . . Yet, the more variation and richness we allow for a concept, the 
more potential we create for disagreement on how it applies to a particular situation, thus 
reducing reliability. 


(p. 158) 


In Figure 3.8, you can see that studies that are strong on reliability but weak on validity 
consistently produce similar results but those results are off the mark regarding what the study 
is trying to illuminate. Conversely, studies that are strong on validity but weak on reliability 
are within the conceptual target but are scattershot, and therefore it is unclear exactly what 
they are illuminating. As you would imagine, low validity combined with low reliability is no 
help in illuminating a topic whatsoever. 


NZ 


Unreliable and Not Valid Unreliable, But Valid 


Reliable, Not Valid Both Reliable and Valid 


Figure 3.8 Reliability validity dartboard 
Source: Nevit Dilmen, https://commons.wikimedia.org/wiki/File:R eliability_and_validity.svg 
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Alas, there is no way out of this inherent tension. Quantitative methods have been critiqued for 
sacrificing validity in favor of reliability, and, indeed, qualitative methods—with their interpretation of 
words and phrases rather than numbers—have a built-in ability to capture a fuller range and depth of 
meanings in concepts. Our best advice for the critical quantitative reader is to be aware of this critique 
and do all that you can to honor a range of meanings in your research and acknowledge the ways in 
which you may fall short. We would also remind the reader that reliability is a key advantage of quan- 
titative methods given its ability to point toward patterns that are larger and more persistent than any 
one experience. Indeed, the emphasis on reliability in quantitative research is one of its chief strengths 
in intervening in social injustices since that emphasis helps encourage research that is verifiable and 
trustworthy. This, in turn, can persuade and compel policy makers, activists, and other agents of social 
change. With a commitment toward validity, the reliable quantitative researcher has much to offer. 


Figure 3.9 Scholar Spotlight: Dr. Bradley J. Bond 


Scholar Spotlight: Dr. Bradley J. Bond 


Bradley J. Bond, Department of Communication Studies, University of San Diego, 
studies media effects related to identity and well-being, primarily among gay, lesbian, 
and bisexual adolescents. He has examined depictions of LGBTQ+ characters and 
celebrities in the media, parasocial relationships, and the role of media exposure in 
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sexual identity development and socialization. His research has appeared in Journal of 
Communication, Communication Research, Media Psychology, Mass Communication and Soci- 
ety, and Journal of Children and Media, among other scholarly journals. 


Q: In your research, what are some of the most important concepts to measure, and how do you 
go about measuring them? 

Dr. Bond: My research investigates the relationships between media exposure, iden- 
tity development, and emotional health outcomes among adolescents, particularly 
teens who identify as lesbian, gay, or bisexual. Outcome variables such as self- 
esteem, social connectedness, and well-being are very important to my work. 
Predicting how these constructs differ in intensity among sexual minority teens 
can shed light on the mechanisms that would help this population better navigate 
their identity development. I live by the motto “adopt and adapt” for many of 
the emotional health variables I measure. That is, I employ previously validated 
measures, but I adapt them in ways that would increase their reliability and valid- 
ity among the populations I study. Por example, Rosenberg’s self-esteem scale has 
been widely accepted as a reliable and valid measure of self-esteem. However, for 
younger populations some items may need to be reworded to ensure that youth 
will comprehend their intent. There is no need to reinvent the wheel, but some- 
times the wheel needs to be aligned—adopt and adapt. 

Q: Are there any concepts that you have found particularly challenging to measure? Why or 
how so? Please give an example. 

Dr. Bond: I find that measurement is more often challenging than not. As social 
scientists, we are often attempting to measure abstract, complex constructs that 
require human interpretation to both measure and interpret. Quantifying iden- 
tity development is an onerous task. Determining the conceptual definition of 
identity complexity or identity satisfaction is deeply important to how we, as 
researchers, then operationalize the variables. This becomes even more difficult 
if we are relying on self-report: Can participants reflect on their selves in such a 
way that allows them to accurately report their identity attributes? Even variables 
that may seem easier to quantify can sometimes require more nuance than novice 
researchers realize. Sexual orientation should be an easy-to-categorize, nominal 
variable: Do you identify as heterosexual, gay, lesbian, or bisexual? However, 
recent research suggests that young people are rejecting these labels; they see 
their sexualities as more fluid and less rigid in definition. How do we capture 
that? How do we quantify a variable required for analysis while respecting the 
population being studied? These are questions I find myself asking as I engage 
in study design. 

Q: How are validity and reliability addressed in some of the measures that you have used in 

your research? 
Reliability and validity are essential to good measurement and, more broadly, 
good science. Reliability is most often addressed through measures of internal 
consistency. If the study design allows for it, test-retest reliability is also common 
in the justification for measures related to identity development and emotional 
health. I always consider the face validity of scales as a starting point, but then try 
to address convergent validity for established measures, and concurrent validity 
when developing new measures. 
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Q: What do you think quantitative methods offer to the study of adolescents’ sexual identities, 
health, and relationships with media, which is the subject of your scholarly expertise? 

Dr. Bond: The benefits of investigating these topics with quantitative methods are 
numerous, but I would argue that the efforts to draw conclusions that are scien- 
tifically objective and generalizable to a larger population are primarily why I’m 
attracted to social scientific methods. Studies on lesbian, gay, and bisexual adoles- 
cent development have historically been conducted from a critical, interpretive 
lens. Though these studies have made remarkable contributions, their samples are 
typically very small and their conclusions cannot speak to generalizable patterns 
or trends within the population. Sexual minority adolescents are a hidden popula- 
tion and, as such, garnering a truly random sample is not practical, so any argu- 
ments about generalizability have to be heeded with caution. However, objective 
analyses of surveys with samples composed of several hundred lesbian, gay, and 
bisexual teens tell us so much about the population, their experiences, and their 
well-being that we did not previously understand, and that may ultimately be 
employed in schools, public health settings, and community groups to help sexual 
minority youth lead happy, productive, emotionally stable lives. 


Social Justice, Measurement, and Avoiding Othering 


You can see from this chapter that the researcher has choices to make when defining and 
operationalizing concepts. Rarely is there only one way to measure a variable. Rather, because 
concepts are typically broad, multifaceted, and complex, the quantitative social scientist has to 
think carefully about how to measure the concept in a way that reflects as much of the com- 
plexity and richness of its meaning as possible but also is able to be recorded reliably within a 
standardized set of values determined by the researcher. This is a sticking point among those 
who find qualitative research and quantitative research to be incompatible. Critics of quantita- 
tive methodology feel that the human experience is too complex, contextually determined, 
and even idiosyncratic to be reduced to the sorts of numbers-as-data measures that quantita- 
tive scholars employ. The fact that quantitative researchers themselves typically predetermine 
the range of values that will be used to measure a concept is also the subject of critique. To 
do so places the researchers in a more powerful, all-knowing position and requires that they 
anticipate any possible response, which is a daunting task since researchers themselves are 
inevitably limited by their own experiences, knowledge, and points of view. 

As we have seen, quantitative research frequently uses categories to make sense of how dif- 
ferent people might have different experiences or views and/or different responses in surveys 
or experiments. For instance, conventional practice in survey research is to include demo- 
graphic questions that measure key features of respondents’ identities, such as their gender, 
race, ethnicity, religion, age, class, sexual identity, or other characteristics. Sometimes these 
factors are at the heart of the research, when, for instance, the researcher is asking research 
questions pertaining to differences that might be explained by these forms of social group 
formations. Yet, other times, survey researchers include demographics so that they can see 
whether relationships between two or more variables at play in their study hold across these 
forms of social group formation. In either case, however, the researcher likely uses categories 
to determine whether participants’ identification with any number of social groups plays a 
part in the phenomena observed. 
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For instance, consider a health communication scholar interested in whether those who 
identify as women might be more persuaded to become organ donors based on the particular 
features of a persuasive media message compared to those who identify as men. That scholar 
may use categories to measure gender identity, placing research participants into groups that 
include, for instance, “cisgender men,” “transgender men,” “cisgender women,” “transgender 
women,” “third gender/non-binary,” “gender fluid,” etc. Or a media scholar may be testing 
whether audiences who vary by race are more or less likely to experience positive mood states 
after seeing the movie, Black Panther. This researcher will need to measure research partici- 
pants’ race in order to test their hypothesis, and they may well employ categories to do so. 
The Research in Action box has more points to consider about the measurement of race and 
ethnicity, in particular. 

Although this may seem logical and valid, there are a number of important considerations 
and even limitations when one uses categories to explore or to describe human experience. 
One important issue is that there is most certainly variation within those that fall into the 
same category so that, for instance, not all Black respondents will feel the same way toward 
the film Black Panther, and not all of those who identify as women will respond the same when 
encountering a persuasive attempt aimed at organ donation. An important pitfall to try to 
avoid, therefore, is engaging in essentialism, assuming there is an underlying, unchangeable 
essence within a category (Medin & Ortony, 1989). Rather, categories can be utilized but 
still considered to be imperfect rather than essential, and variation within categories can be 
included in the discussion of results, as well. 

Another important issue is that we human beings are, of course, more than just the result 
of our experiences within one or more forms of social categorization considered individually. 
Rather, we have a number of differing ways of seeing ourselves that all simultaneously coexist, 
so that there are multiple, “intersectional” aspects of who we are. Intersectionality is a term 
that has long been at the heart of feminist and critical race theory, particularly within Black 
feminist writing that has spoken to the multiple oppressions of gender and race experienced 
by women of color (Collins, 2000; Crenshaw, 1989, 1991). Cole (2009) defined intersection- 
ality as “analytic approaches that simultaneously consider the meaning and consequences of 
multiple categories of identity, difference, and disadvantage” (p. 170). There is evidence that 
scholars across a number of fields in social science are thinking through the ways in which 
elements of an intersectional analysis might be advanced using quantitative methods (Choo & 
Ferree, 2010; Cole, 2009; Else-Quest & Hyde, 2016). Applying these ideas to quantitative 
research within Communication, the second author of this book has theorized about the 
Critical Media Effects Framework, which argues for a more nuanced understanding of identi- 
ties and power relations within media effects scholarship (Ramasubramanian & Banjo, 2020). 

Although categories themselves may be viewed as too artificial and too reductionist by 
some scholars studying intersectionality, it is possible to use multiple categories in quantitative 
research and recognize that they are intertwined (McCall, 2005). Yet, most agree that simply 
identifying and employing multiple categories—like gender and race—in research is not suf- 
ficient to position that research as intersectional. Rather, identifying ways that social structures 
and social contexts perpetuate or challenge the power positions and inequalities experienced 
must also underlie the analysis (Else-Quest & Hyde, 2016). 

Another pitfall that the quantitative social scientist may fall into is using categorization to 
compare a dominant group to a marginalized group, which—without proper consideration 
of context, history, and social, economic, and cultural forces that help create and maintain 
those positions of dominance and marginalization—is too easily used to reinforce rather than 
work to dislodge inequalities. As we have seen in Chapter 1, the use of a “deficit discourse,’ in 
which the challenges of those facing oppression are the central focus and in which outcomes 
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of those in more privileged positions are juxtaposed against those in more marginalized posi- 
tions, can serve to reify White and patriarchal dominance (Solórzano & Yosso, 2001). 

With all of these challenges, then, how does one collect quantitative data with a critical 
consciousness? There are no easy answers, but there are some general suggestions. 


e First, we suggest that researchers spend time with the communities they hope to study 
and listen to and learn from the members of those communities. This will check the 
researcher’s ego, balance the researchers own subjectivities and understandings of con- 
cepts and phenomena, and help come up with language that can be used in the framing of 
variables and their values or attributes that will be more authentic to research participants. 

e Second, and somewhat similarly, conducting a pretest of measures to be used in a study 
among even a small group from the population of interest will similarly help identify 
places in which the researcher has fallen short in responding to the range of understand- 
ings and experiences reflected in the operationalization of concepts. A pretest is an 
initial inquiry into a research area that is usually less formal and smaller in size or scope 
compared to the ultimate study. Findings from a pretest can be used to shape the ultimate 
study. 

e Third, researchers should consider employing some open-ended items to measure some 
aspects of identity such as gender identity, sexual orientation, and race or ethnicity. In 
doing so, there is no need to anticipate every possible response, and the research par- 
ticipant will not have to experience the feeling of looking in vain for an option that 
corresponds to their own identity. When we allow respondents to use their own words 
to describe their identities, the catch-all option of “other” is not necessary and there- 
fore will not run the risk of offending or marginalizing participants. Open-ended data 
can later be placed into categories, as needed for statistical analysis, as discussed in this 
chapter. 

e Fourth, researchers should consider whether research participants feel safe in participat- 
ing in the study in addition to whether they feel respected and recognized in interactions 
with researchers and in completing the study. Measuring variables related to inequalities, 
oppression, the silencing or the amplification of voice, and the distribution of privi- 
lege and power is not only complex but also potentially quite personal and sensitive. 
Researchers have a responsibility to make sure research participants are provided with 
the conditions needed to respond to variables tapping some aspect of these concepts in a 
way that is safe and secure. As we have seen in Chapter 2, the principles and practices of 
informed consent help address some of the issues of assuring research participants a safe 
space. 

e Finally, researchers must always be critical of themselves and identify and acknowledge 
their own limitations. There is no such thing as a perfect piece of social justice scholar- 
ship. But, as we read in Chapter 1, there is definitely a space for researchers to continually 
engage in the reflexive process of attempting to design and carry out their studies with 
integrity and a critical consciousness. 


Summary 


In order to conduct research with the goal of working toward a more just and equitable soci- 
ety, variables need to be assessed for whether their conceptualization responds sufficiently to 
multiple ways of knowing and experiencing the world. In coming up with ways to define 
concepts and develop the options used to measure them, critical quantitative researchers must 
do their best to avoid assuming there is a single shared understanding or that there is equal 
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access to the structures of power. We must remember that the identities of the people who 
we study are complex and their experiences play out in a broader historical, cultural, social, 
economic, and political context that we should carefully consider as social scientists. 

Baked into the methods of quantitative research are processes and procedures in place to 
guide the researcher in creating data that are valid and reliable and therefore are convincing 
enough to help make the case for social justice and social change. Constructing variables in a 
manner that is precise, accurate, clear, and meaningful goes a long way in providing data that 
people can trust. In order to do so, researchers must be sure their operationalizations of vari- 
ables meet the standards of solid research design, such as by providing options that are mutu- 
ally exclusive and exhaustive and in carefully considering levels of measurement. 

Researchers will want to consider the full range of meaning in the concepts they are seek- 
ing to measure, including whether those concepts have important dimensions to capture or 
could be best identified through various indicators. At the heart of creating new knowledge 
using methods from the quantitative toolkit is the balancing of validity and reliability in the 
research design. Carefully considering both reliability and validity will go a long way in ensur- 
ing that the research you are producing is meaningful as well as sound. Reliable and valid 
data are needed to challenge the structures and policies that keep some in power and some in 
oppression. In providing data that are both reliable and valid, quantitative researchers have a 
great deal to offer to policy makers, activists, and organizations working toward social justice. 


Discussion Questions and Activities 


1. What concepts are you interested in exploring in your own research? Identify at least one 
concept and then brainstorm two variables that would stem from that concept. Finally, 
come up with an operational definition of each variable. How, exactly, can your concept 
of interest be measured? 

2. Is there any concept that you think is completely impossible to measure using quantitative 
methods? What is it? If you push yourself, can you think of at least a partial way to get at 
that concept through operationalization? 

3. How might we measure the concept of prejudice? Come up with two or three ways you 
might be able to measure the level of research participants’ prejudice on a survey by gen- 
erating both the language for the survey items and the response options. Then, determine 
the level of measurement for each measure you generated. 

4. Why can there be tension between validity and reliability in quantitative research? Explain 
this in your own words and think of an example that illustrates the tension. 

5. Why are conceptualization and measurement so important in producing research for 
social justice? 
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4 Sampling and Representation 


Through social research, we can provide a space for people to lend their voices and be heard. 
In research that involves people as participants, the researcher typically will decide who will be 
asked to participate in the study, thereby determining the group that provides the data. When 
research involves getting input from people, those chosen to constitute the research sample 
are given voice. They have the chance to have new knowledge built from their experiences 
and points of view. Research participants have direct representation in that they can speak for 
themselves when filling out questionnaires or otherwise providing data. And when they are 
selected carefully, as you will see in this chapter, research participants have the potential to 
indirectly represent others from the same population who do not directly participate in the 
research. 

We saw in Chapter 3 that a single way of measuring a complex variable is likely to fall 
short of capturing the complete set of understandings and meanings of concepts that exist in 
the social world. Similarly, no one group of research participants can fully capture the entire 
range of experiences and points of view that operate in the wider group from which those 
participants are selected. However, as you will read in this chapter, there are ways to select 
research participants that do the best job possible of those participants standing in for the 
larger group from which they were selected, even if that ability to represent the larger group 
is not perfect. Carefully chosen research samples, then, can be used to extrapolate to a wider 
body. And when particular sampling procedures are followed, the idea of giving voice is 
amplified beyond those directly studied into larger populations, a unique strength of quantita- 
tive research for social justice. 

Sampling is the process of “deciding what to observe and what not to observe” (Bab- 
bie, 2010, p. 189). It sounds so straightforward, and in many ways it is. But, just ike with 
any researcher-originated decision, there are a number of key considerations and important 
implications when deciding how to draw a research sample. Thankfully, as we have seen 
with the process of operationalization, there are widely accepted guidelines, principles, and 
approaches, many of which are established through applying the principles of the scientific 
method. In this chapter, we will introduce those guidelines, principles, and approaches as well 
as consider the social justice implications of each. 

The question of who or what will comprise the research sample is vitally important in 
quantitative social science research. In fact, it is one of the central elements of what we have 
called “the power of numbers.” One of the chief strengths of quantitative research 1s its ability 
to explore patterns that are determined to be widely shared—or, conversely, are shown to be 
relatively rare—within a large and potentially heterogeneous group. As we saw in Chapter 1, 
qualitative research typically uses small samples and is not always concerned about general- 
izing findings to a larger population. Qualitative research, therefore, can tell us a whole lot 
about a relatively small body of people or of texts. But those observations may well be quite 
unique to that small group, and therefore their applicability to a broader body can be limited. 
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In quantitative research, we draw somewhat less detailed observations from a larger body of 
people or of texts. In doing so, we strive to say something more widespread and more gen- 
eral about the phenomenon we are studying. That is a major opportunity for social scientists 
attempting to understand and to try to change the world around them. 


Samples and Populations 


Almost all social science research involves sampling. If you are interested in surveying adoles- 
cents to ask whether they have encountered bullying on social media because, for instance, 
their gender expression does not conform to “traditional” gender roles and norms, you have 
to determine which adolescents to survey. If you want to study whether people experiencing 
threats to their mental health are depicted in popular films as violent and dangerous, you have 
to decide which films you will analyze to explore this question. Clearly, it is not feasible to 
study all adolescents who exist on the planet or even all films that have ever been made, and 
therefore you must decide which subset you will study. 

If you could study all adolescents or all films, that is known as capturing the popula- 
tion in your research, with population defined as the entire set of research units of interest. 
Quantitative researchers typically want to draw conclusions about the population. Defined 
as “the totality of people or elements to be represented in the sample” (Anderson, 2012, 
p. 164) or “an entire set of objects, observations, or scores that have some characteristic 
in common” (Wrench, Thomas-Maddox, Peck Richmond, & McCroskey, 2008, p. 216), 
the population includes every single potential member of the body of people or of texts 
the researcher is seeking to observe (Figure 4.1). If you can gather data from the entire 


Population 


Figure 4.1 Population and sample 


Source: Stephen Warren 
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population, sampling from that population is not necessary. Researchers who study the 
entire population of interest in their research have taken what is called a census rather 
than a sample. 

Although it is rare in social science to be able to make observations about the population 
in its totality, it is not unheard of. If, in the words of Wrench and colleagues (2008, p. 216), 
the “characteristic in common” that defines the population is somewhat narrow, it is possible 
to study the whole population. For instance, maybe you are interested in how the decision to 
withdraw U.S. troops from Syria was covered in the New York Times in the month following 
the announcement. You could certainly capture all such news items in the Times to make your 
observations. Or, if you go to a rally against hate speech on your own local college campus 
and are interested in finding out what inspired people to attend, you could conceivably survey 
all those present at the rally. 

For many studies, however, constraints involving lack of access, time, or funding make 
it difficult or even impossible to capture the entire population, and therefore the researcher 
must decide how to go about sampling. For quantitative research, one of the chief goals is 
to establish the generalizability of findings, or the ability of the observations made from 
the sample to closely match observations that would appear if the researcher had been able to 
study the entire population. It is important to be able to say that observations made within a 
chosen sample are likely to reflect, relatively accurately, the larger population from which the 
sample was drawn. It is precisely these observations of the wide social world that quantitative 
researchers try to make. 


The Representativeness of Samples 


Establishing the ability of a sample to generalize to a population is closely related to the con- 
cepts of validity and reliability that we explored in Chapter 3. As we have seen, quantitative 
research seeks to be both valid and reliable. It seeks to be valid in that the data it collects 
should tell us something meaningful and accurate about the phenomenon being studied. And 
it seeks to be reliable in that results should be able to be produced with a fair degree of con- 
sistency if the study were to be repeated. 

One important form of validity is external validity, or the degree to which results 
can be generalized beyond the scope of the individual study. If you find a result in one 
sample, in order to conclude that the result is meaningful and accurate, you should be 
able to find the same result (more or less) in additional samples from the same popula- 
tion, as well. This is because a study with strong external validity is able to observe 
something that would actually be occurring broadly in the wider world rather than only 
within the study itself. A study has external validity if the observations made in the study 
apply beyond the specific circumstances—the people, the timing, the setting, etc.—in 
which it was examined. 

Consider a study, for instance, that finds that, within a sample of college students, the 
higher the students’ levels of empathy, the stronger their emotional responses to news cover- 
age of war. It would be important to test whether that same relationship between empathy 
and emotional responses appears within another sample of college students before we can 
make decisive conclusions. Better yet, we could see if the observed relationship between the 
two variables (empathy and emotional responses) extends beyond students to a non-student 
sample that is more heterogeneous in age and education. If it did, a quantitative scholar would 
feel more confident concluding that empathy and emotional responses to news coverage of 
war were related not just in research studies but more broadly in the “real world.” This is the 
essence of external validity. 
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Figure 4.2 Scholar Spotlight: Dr. Muniba Saleem 


Scholar Spotlight: Dr. Muniba Saleem 


Dr. Muniba Saleem is an associate professor in the Department of Communication 
at University of California, Santa Barbara and a faculty associate at the Institute for 
Social Research at the University of Michigan. She studies how media influences 
interpersonal and intergroup relations. Her work on media influence on the ethnic and 
national identities of immigrants, acculturation, and immigrants’ trust in U.S. govern- 
ment and relations with majority-group members has been published in some of the 
top Communication and Psychology journals in the field. Dr. Saleem has been called 
upon to be an expert source for both local and national news outlets, including NPR 
and the Huffington Post. Her research (co-authored with Dr. Srividya Ramasubrama- 
nian, the second author of this book) was cited and discussed in amicus curiae briefs that 
were submitted to the U.S. Supreme Court and lower courts in support of the plaintiffs 
in Hawaii v. Trump, 585 U.S. _ (2018) opposing the executive order often referred to 
as the “Muslim Ban.” 


Q: What factors guide sampling decisions in your research? How do you strive for generaliz- 
ability when drawing a research sample, for instance? 

Dr. Saleem: The overall goals of the research and the design are important in guiding 
my sampling decisions. For instance, if I am interested in testing theoretical prop- 
ositions of a model and I expect to use random assignment to compare differences 
between groups, I would be fine with a student or MTurk sample.' If however, the 
research question is more applied and is testing a real-world phenomenon and/or 
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is using cross-sectional or longitudinal designs, I would prefer a more representa- 
tive sample so the findings can be generalizable. 

Q: What are some of the opportunities and challenges you’ve encountered in sampling? 

Dr. Saleem: I have worked with majority and minority racial groups in my research. 
Overall, working with majority group samples is relatively easier than work- 
ing with minority groups. This is because these samples can easily be acquired 
through college subject pools, MTurk, and/or by hiring companies that can find 
representative numbers on the specific criteria you desire (e.g., political orienta- 
tion). In addition, such work is often perceived by reviewers to be more generaliz- 
able and representative of general human behavior. Working with racial minority 
samples is difficult given that they are not readily available in college subject pools, 
Mturk, and through research polling companies. Getting access to this sample 
requires networking and searching for organizations through which you can reach 
out to this sample. Given the added effort and time required in recruiting, the 
cost of a research project involving racial minority samples is often more than the 
cost of a sample not requiring racial minorities. Even if you get access to racial 
minorities, they are often suspicious of whether the research is being used in a 
manner that targets them individually or as a group. This can lead to low recruit- 
ment, attrition, or lower quality data. Another challenge I have encountered is 
that reviewers are often more critical of research using racial minority samples 
than research that does not use racial minorities. This is, unfortunately, due to a 
lack of knowledge about the studied group; questions surrounding the justifica- 
tion for why the specific, relative to another, minority group was chosen; and/or 
inquiries regarding how generalizable the findings found using a racial minority 
sample are to the “general population.” 

Q: One challenge for quantitative researchers is to obtain sufficiently large samples of non- 
dominant or minoritized groups. What are some of the approaches you have used to include 
members of non-dominant groups in your studies? 

Dr. Saleem: Many of my recent research questions are focused on racial minority 
samples. I have made extra effort to engage with the communities I am interested 
in studying. For instance, it is important to reach out directly to community lead- 
ers and organizations that cater to minority groups to establish communication 
and trust. Researchers also have to make an honest effort to understand the issues 
and concerns that are important to the community they aim to recruit from. This 
entails attending student and community events affiliated with the group, hearing 
the perspectives of community members, discussing the importance of research 
for the group, and offering to share and learn from the findings of the research 
study. 

Q: Why do you think quantitative methods, in general, are important for studying the issues 
and topics you take up in your research? 

Dr. Saleem: I appreciate that the data are objective and the findings that emerge are 
not subject to my interpretation or bias. The potential to discuss causation from 
experimental designs is another advantage. Finally, the ability to meta-analyze 
multiple studies on the same topic can uncover generalizability and limitations of 
a particular finding. 


! MTurk is an abbreviation for Amazon’s Mechanical Turk crowdsourcing marketplace platform that research- 


ers have used to pay individuals to participate in or assist with their research. 
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One way of attempting to achieve generalizability of findings is carefully considering the 
degree to which a single sample used for a study is representative of the larger popula- 
tion from which it is drawn. Are the college students in the sample described in the earlier 
hypothetical study clustered around particular majors like Journalism or Communication that 
might already be sensitized to the topic of news coverage of war? How do the demographics 
of the student research participants in the study match up with the demographic composi- 
tion of all of the students at the college from which the sample was selected, or even college 
students in the U.S. (or any location) in general? The stronger the claim that the sample used 
for the study generalizes to or reflects the population as a whole, the stronger the external 
validity of the study. 

Although generalizability and representativeness are sometimes used interchangeably, it is use- 
ful to think of representativeness as applying to samples and generalizability as applying 
to findings (Gobo, 2004). Representative samples are generally assumed to produce gen- 
eralizable findings, but such an outcome is not a guarantee. Even if you have a representa- 
tive sample that does an excellent job of standing in for the larger population, you may not 
always produce generalizable findings that reflect what is actually present in the population 
(Figure 4.3). This is because other things can go awry, such as threats to validity associated 
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Figure 4.3 Sampling and the concept of representativeness 


Source: Martyn Shuttleworth, Lyndsay T Wilson (Nov 15, 2008). What is Generalization?. Retrieved Jun 03, 2020 from 
Explorable.com: https://explorable.com/what-is-generalization 
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with operationalization or errors or biases in how participants respond to items and provide 
data. In order to know for sure whether results from a sample can generalize to the wider 
population, one would have to study the entire population. That, as we have seen, can be 
quite difficult if not downright impossible if the population is too large or too difficult to 
capture in a census. 

Nonetheless, the goals of obtaining representative samples and generalizable findings are 
exceptionally useful when researchers want to make a case that what they are finding is not 
the result of a sample selection process but rather is a broader phenomenon likely to be found 
in the wider world. Quantitative researchers employing sound strategies to address representa- 
tiveness and generalizability, for instance, cannot be accused of “cherry picking” only those 
cases that would support a particular conclusion. Imagine that a hypothesis is supported in a 
sample that was not chosen for any pre-existing connection to the phenomenon being stud- 
ied. This strengthens the claim that the phenomenon is widely experienced. For social justice 
research, in particular, quantitative analysis can be used to “bolster the impact of small sample 
qualitative findings by documenting the ‘representativeness’ of oppressive conditions in larger 
representative samples” (Ponterotto, Mathew, & Raughley, 2013, p. 48). Indeed, this type 
of research can show that an experience of oppression—or, conversely, of dominance and/ 
or privilege—is widely shared across members of a particular social group, which is a vitally 
important observation for social justice. 


Probability Sampling 


The primary way to ensure that a sample is representative of the population that it seeks to 
reflect is to use probability sampling. In probability sampling, every unit of the population 
has an equal chance of being selected to comprise the sample. This is accomplished by using 
random selection. If each unit comprising the sample is chosen randomly—meaning, once 
again, that each potential unit has an equal chance of being selected—the researcher can avoid 
bias or undue influence on the sampling procedure that might result in the sample being 
skewed in some manner. 

Research methods guru Earl Babbie (2010) reminds us that, “In connection with sampling, 
bias simply means that those selected are not typical or representative of the larger populations 
that they have been chosen from. This kind of bias does not have to be intentional. In fact, 
it is virtually inevitable when you pick people by the seat of your pants” (p. 197). Biases can 
seep in consciously or unconsciously. In either case, the result is a sample that does not reflect 
the population, which threatens generalizability of findings. 

The researcher drawing a random sample has a number of options to use. One common 
method of probability sampling is the simple random sample (Figure 4.4). In order to 
draw a simple random sample, the researcher must have an exhaustive list of every unit in the 
population, also known as the sampling frame. The units in that list are assigned numbers (if 
they do not already have them), and a random number generator selects numbers within that 
range to constitute the sample. For instance, to obtain a simple random sample of students on 
a college campus, a list of all currently enrolled students could be obtained from the registrar 
and then numbered. If there are, for instance, 10,000 student names, the researcher could pro- 
gram a random number generator to provide 100 random selections between 1 and 10,000. 
The students assigned to each of those 100 numbers would then comprise a simple random 
sample of 100 students that could be used in research to represent the larger student body. 

Wondering how to access a random number generator? It used to be routine practice to 
print a table of random numbers in quantitative research methods books or statistics text- 
books for many decades, and some of them still contain such a feature. Yet, the modern 
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Figure 4.4 Simple random sampling 


Source: Stephen Warren 


researcher can also use one of many free online programs to generate random numbers 
within defined ranges, as well. As of this writing, random number generators can be found 
online at https://andrew.hedges.name/experiments/random/; randomnumbergenerator. 
com; or randomizer.org. 

A variation on the simple random sample method is the stratified random sample. This 
technique is useful when the population can be divided into subgroups (“strata”) that are 
distinct from one another and the researcher wants to obtain observations from each group. 
In the case of the students on a college campus, for instance, perhaps you have a research 
question in mind for which it is necessary to compare first-year students with seniors. You 
could stratify the sample by obtaining a comprehensive list of each (a list of first years and a 
list of seniors), and then you could draw two separate simple random samples from each list. 
Together, you would have the sample you need to study similarities or differences between 
first-year students and seniors. In the process of selection, each of the first years and each of 
the seniors, in turn, had an equal chance of selection for inclusion in the sample. 

Importantly, to draw a stratified random sample, you need to ensure the proportion of 
the subgroups in the sample is the same as the proportion of subgroups in the population 
(Figure 4.5). For instance, if there are 3,000 first-year students and 4,000 seniors in the popu- 
lation and you want to create a stratified random sample that is 10% of the population, you 
would need to select 300 first-year students (10% of the total 3,000 first-year students) and 
400 seniors (10% of the total 4,000 seniors) to include in your sample. 

Another probability sampling technique that is quite commonly used by Communica- 
tion researchers and other social scientists is systematic sampling, which is often easier to 
accomplish than simple random sampling. It can get quite tedious to obtain as many as 100 
or more random numbers as is necessary for a simple random sample. In systematic sampling, 
once you have obtained a comprehensive list of population units, you choose an interval at 
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which to select units from the list rather than using random numbers. You can determine the 
interval based on the total number you would like to wind up with for your sample. 

If you want to study about 200 comments posted to a controversial YouTube video at one 
particular point in time, for instance, and that video has 2,422 comments when you are doing 
your analysis, you could choose every 10th comment to study and end up with a sample of 
242 comments. Ten would be the interval that you used to obtain your systematic sample. Or, 
if you had a list of 4,000 members of gay-straight alliance groups from high schools around 
the country, you could choose every 15th person on the list to derive a systematic sample of 
266 members. Figure 4.6 shows a systematic sample in which the researcher chose a starting 
point (in this case, the second unit) and then chose every third unit. 

These examples bring up a related term that is important to introduce. The sampling 
ratio is determined by dividing the sample size by the size of the population. A sample of 
200 YouTube comments from the total of 2,422 comments, for instance, has a sampling ratio 
of .08 (200 divided by 2,422). A sample of 266 members of the total 4,000 members of high 
school gay-straight alliance groups has a sampling ratio of .07 (266 divided by 4,000). If you 
have a particular goal in mind for the sampling ratio, you can use that to determine what 
interval should be used in a systematic sample. 

As long as the list has no particular pattern in it, systematic sampling works efficiently and 
well. Beware of whether the list does have a pattern. For example, consider a list of college 
students that for some reason was organized by listing first a first-year, then a sophomore, then 
a junior, and then a senior, and then repeating that pattern. If you selected every 3rd student 
on the list, you would end up with an entire sample of juniors rather than having a broad 
distribution across class years. 

Another type of probability sampling is cluster sampling also known as multistage sam- 
pling, used when there is both a population and a subpopulation of interest, with the sub- 
population nested within the population. Such sampling typically involves two steps, which 
is why it is often called multistage sampling. First, the main population elements are sampled. 
Then, one or more subpopulations within that resulting sample are sampled. For example, if 
you were interested in studying opinions among residents of a selected city, you could begin 
by listing all blocks in the city and then randomly select some blocks. Once you have selected 
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Figure 4.6 In systematic sampling, every nth item is selected from the population 


Source: Dan Kernler, https://en.wikipedia.org/wiki/File:Systematic_sampling. PNG 
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those blocks, you could determine the total number of households on each selected block and 
then randomly select households. The households are clustered or nested within blocks in this 
example, and thus the sampling process takes two steps. Researchers use this approach to sam- 
pling when there are different levels of populations of interest, like a subgroup within a group, 
and therefore the sampling approach calls for making selections in more than one stage or step. 


Nonprobability Sampling 


Despite the benefits of probability sampling for adhering to the principles of the scientific 
method and ruling out bias in sample selection, many times obtaining a probability sample is 
simply not possible. Our hypothetical list of 4,000 members of gay-straight alliances in high 
schools across the country is an example ofa list that would be exceptionally difficult to obtain. 
For the occasions when you simply cannot take a random selection from a comprehensive 
list, there are a number of nonprobability sampling techniques used quite commonly in 
Communication and other social sciences. Carefully constructed nonprobability samples can 
still lend important insights into the social world, so do not despair if your research takes you 
to this set of sampling approaches. Yet, a good principle to uphold is that when you can use 
probability sampling techniques in quantitative research, you should do so. 

One common approach in which the researcher samples from a population in a non- 
random manner (in which potential members of the sample do not have an equal chance of 
selection) is a convenience sample, composed simply of what or who is available. If you 
are interested in exploring what people think of a Massachusetts law that protects the right to 
use a bathroom or locker room that corresponds with one’s gender identity, for example, you 


Do you believe in 
global warming? 


Figure 4.7 Convenience sampling is a type of nonprobability sampling where the items are selected based on 
ease of reach and access 


Source: Emily Riewestahl 
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might ask a group of people to which you have access to fill out a survey on the topic. A col- 
lege professor, for instance, could ask some students on campus. A college student doing their 
own research could ask all of the members of a residence hall who they encounter in a single 
day. A person could stop folks at the mall or in the center of town or any other busy place 
and ask them. The same convenience might apply to media texts rather than human research 
participants. Por example, if you want to know how many Asian or Asian American actors 
appear in primetime television programs, you could program your DVR one night to record 
as many primetime shows as you can and examine the casts of the programs you capture. 

It should be clear that samples such as these are inherently quite limited in their abil- 
ity to represent a larger entity, and therefore findings from studies that use convenience 
samples are not generalizable to wider populations. All sorts of biases are likely to occur in 
the convenience sample examples we listed, both conscious biases and unconscious ones. 
What if the college professor polling students about the law regarding gender identity and 
bathrooms/locker rooms does so by approaching students in front of a building where only 
engineering majors take classes? The professor might only get the views of students within 
that one major rather than those from a broader cross-section of majors. What if the single 
night of programs recorded on the DVR was different from the other nights of primetime 
programming in some relevant way, like it happens to have more or less Asian or Asian 
American actor presence? These are just a couple of many limitations to the scenarios 
listed here. Fundamentally, convenience samples simply cannot accurately reflect the larger 
population of interest. 

A nonprobability sampling option that is likely to be superior to convenience sampling (but 
still inferior to probability sampling) is purposive sampling. In purposive sampling, samples 
are selected because the researcher has logically determined that they reflect the characteristics 
that form the basis of the research. This type of sample is useful when there is “a small subset 
of a larger population in which many members of the subset are easily identified, but the 
enumeration of them all would be nearly impossible” (Babbie, 2010, p. 193). In other words, 
you know who (or what) you would like to study and you can access some members of (or 
units in) that group of interest, but you cannot obtain a comprehensive list of all members of 
(or units in) the group. 

For example, if you are interested in whether adolescents experience body image distur- 
bances from seeing beauty ideals presented through social media, you could visit a local high 
school and secure permission to survey a few classes. The classes are chosen strategically to 
yield adolescents of the age you have determined to be of interest for the study, but there is no 
random selection involved in selecting either students or classes that would provide the data. 
In a content analysis example of purposive sampling, Susan Mello (2015) set out to study the 
health information that new and expecting mothers were receiving about environmental risks 
to infants’ health. Mello first informally polled new mothers and mothers expecting a child to 
find out where they were most likely to receive health information. From their identification 
of sources (magazines, websites, and general news), Mello selected content from “parenting 
websites with the highest traffic rates during the prior year. . . , parenting magazines with the 
highest circulation rates during the prior 6 months” (p. 1247) and news stories found in the 
database LexisNexis over a 6-month period. The selection of this content was purposefully 
chosen, as you can see, but it did not contain a random selection element. 

A third type of nonprobability sampling is quota sampling, an approach that starts with 
the identification of attributes of the population of interest (like percentage of the popula- 
tion that are women and/or people of color) and then selects participants (without a random 
selection process) until the sample contains the same proportions of those attributes as the 
population does. In other words, researchers set a quota for a desired number of participants 
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who meet the attribute they are focusing on and then invite participants to the study until 
that quota is met. 

Imagine you are interested in testing whether Black football fans in the U.S. are more likely 
to agree with players kneeling during the national anthem in support of the Black Lives Mat- 
ter movement compared to White football fans. You may decide to set a quota in which 50% 
of the sample identifies as Black and 50% identifies as White. To engage in quota sampling, 
you keep inviting people to participate in your study until that 50-50 distribution is achieved. 
If this sounds like stratified sampling to you, you are correct. Quota sampling is just like strati- 
fied sampling except that the sample is not chosen at random. 

The first author of this book used quota sampling in her research by employing the research 
company Qualtrics to obtain a sample that reflects the racial and ethnic distributions found 
in the U.S. census (Scharrer & Blackburn, 2018). Participants signed up for and took the 
study until the predetermined quotas for race and ethnicity were met. The approach has the 
advantage of including people of color in the sample in equal proportions to their population 
in the USS. 

Finally, snowball sampling is another method of nonprobability sampling in which 
the same purposive choices as named earlier identify one or more relevant members of 
the sample and then those members, in turn, suggest others in a fashion that “snowballs” 
into a large sample. Once you have one research participant agree to be in the sample, for 
instance, that participant can put you in touch with others they know, and so on and so 
on until a sample is achieved. This approach can be useful in studying populations that are 
hard to reach (like members of the antifa movement, for instance) but, as is the case with 
the other nonprobability sampling methods explained here, simply cannot represent the 
entire population. 

There may be multiple reasons why populations are hard to reach. A hard-to-reach popula- 
tion may be small in size, difficult to locate or identify, and/or may be stigmatized. Snowball 
sampling is quite common in social justice research, particularly when gathering data from 
stigmatized groups. For instance, Liang Chen, Yu Guo, and Jingyuan Shi (2019) used snow- 
ball sampling to sample gay men living with HIV/AIDS in China to study their use of social 
media as a means of social support. 


Getting Mathy 


We have established that probability samples are superior to nonprobability samples in quanti- 
tative research, but we have not yet shared the details about why that is the case. The answer 
can be found in probability theory. We find that an apt metaphor for how probability theory 
works with sampling is to think of a box of multi-colored marbles that contains the entire 
population of interest. You reach into that box of marbles once to grab a handful, and that 
handful comprises your sample. What are the odds that your handful reflects the same dis- 
tribution of colors (the same number of blue, yellow, and red marbles, for instance) as were 
present in the population—the entire contents of the box—as a whole? What if you did the 
same thing again? How likely is that second handful to reflect the original proportions of 
blue, yellow, and red marbles? Each handful is like a sample in research and the entire box of 
marbles is the population. 

One major advantage of employing random sampling techniques is that doing so allows you 
to apply probability theory to formally estimate how likely your sample is to have captured 
what is going on in the larger population. In order to proceed with the logic and a bit of the 
math behind these processes, it is necessary first to define parameters and statistics. A param- 
eter is what a variable would look like if you were able to know how it exists fully in the 
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Figure 4.8 Types of sampling methods 


Source: Stephen Warren 


entire population. A statistic is what the variable looks like in the sample you have drawn 
for your research. Now we will talk about how to determine the distance between those two 
things—how likely is it that the statistic matches the parameter? The difference between the 
population parameter and the sample statistic is sampling error. 

With probability samples, you can actually use some basic mathematical principles to 
estimate sampling error, so that you know just how likely your results are to stand in well for 
a larger population. Nonprobability samples, alas, simply do not have the necessary quali- 
ties to allow for these calculations. What are those qualities, you ask? The most important 
quality is that for probability samples, the central limit theorem shows that taking mul- 
tiple samples from the population—and thereby deriving multiple statistics on a variable of 
interest—will result in statistics that cluster around the mean of the population parameter. 
Once enough samples are selected from the population, those statistics will be distributed 
in a bell-shaped curve with what we call a normal distribution, clustered around the 
mean of the population and then tailing out from that center point on each side, as seen in 
Figure 4.10. 

A real-world example should help illustrate this point. Assume that we sample 100 women 
residents of a local town and we ask them a yes or no question: Have you ever experienced 
sexual harassment on the job? In that sample, we yield a statistic that shows 38% answered 
in the affirmative. We now sample another 100 women, and this time 45% say they have 
been sexually harassed. We sample again, with 200 participants this time, and 40% say yes. 
Next time, among 85 participants, 39% say yes. If we take a smaller sample, say just 30 or 40 
women, the statistic might vary from the estimates established in the larger samples a bit more 
widely. We might get a result of 22% or 59% answering in the affirmative, for instance. Etc. 
etc. This process goes on. 
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Figure 4.9 Probability textbook 
Source: “Photo 221/365: My favorite book #edugood.” buistbunch, www.flickr.com/photos/63875226@N04/9476936946 


If we plot all of those results—each of the statistics—together in a graph, the central limit 
theorem shows that those multiple sample statistics that we have drawn will be arrayed in 
such a way that they will cluster around the population mean and will be distributed in a 
normal, bell-shaped curve around that mean. The population mean is what we would have 
found had we been able to study all women residents of the local town. The statistics from 
the larger samples will do a better job of bunching up in the center of the curve, whereas 
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Normal Distribution Curve 
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Figure 4.10 Normal distribution curve 
Source: M. W. Toews, https://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg 


the statistics in the smaller samples may end up in the tails of the bell-shaped curve. In our 
example, if the statistics from all of those samples end up clustering around 41%, we can esti- 
mate that 41% was the value of the population parameter. In other words, we can reasonably 
assume in our example that if we had been able to study the entire population of women 
residents of the town, we would have found that 41% reported having experienced sexual 
harassment on the job. 


Having “Confidence” in Your Sample 


Even with probability samples, sampling error depends not just on sample size but also on 
how much variation there is in the population. It stands to reason that the more consistent 
(the less varied) the population parameter, the more likely any given sample would be able 
to approximate that population parameter. The Pew Research Center estimated in 2017, for 
instance, that 81% of Black respondents in its national poll considered racism to be a major 
problem in society (Neal, 2017). If you were to draw samples of Black respondents, given 
how widespread this view is within that population, chances are you would find estimates 
of agreement that racism is a major societal problem that correspond fairly closely to such a 
widely held view among the overall population. Sampling errors will be smaller, therefore, 
when populations are more homogeneous on the question of interest. 

When you calculate sampling error, the estimate you come up with is called standard 
error. Standard error is a mathematical function of the distribution of the population parame- 
ter (which accounts for variation as we have explained) and the sample size. For a variable that 
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has just two values—like our yes or no question to women about having experienced sexual 
harassment—standard error can be calculated by taking the square root of the product of the 
distribution of that statistic (the proportion of women who said yes times the proportion who 
said no) divided by the sample size. Let us go back to our example in which we estimated 
41% of women reported sexual harassment on the job, leaving 59% who had not, and plug 
in a sample size of 100. The formula for standard error for this illustration, then, would be 
the square root of .41 x .59 divided by 100, which equals .049, which we can round to .05. 

Computers can (and will!) do this math for you, so there is no need to memorize the for- 
mula or even, frankly, to run through an example of estimating standard error yourself. In fact, 
when variables have more than just two values (which many do), the math gets exponentially 
more complicated. Yet, it is important to know the logic behind how researchers can make 
claims about the odds that their samples have captured the population parameter. In order to 
do so, we have to add one more layer of complexity to this logic. 

Statisticians have determined that if multiple samples are taken at random from a popula- 
tion—as we have also explained—34% of those sample estimates will fall within one standard 
error below the population parameter, and 34% will fall within one standard error above the 
population parameter. For our example, then, for which we calculated a standard error of .05, 
which is also known as 5%, we would add that 5% to the estimated result of 41% and come 
up with 46%. We would also subtract 5% from 41% to account for the other side (below) the 
population parameter and come up with 36%. Together, then, on both sides of the population 
parameter, 68% of the samples (34% times 2) we could take from the population will yield 
estimates between 36% and 46%. 

Statisticians have also determined that 95% of the samples we could randomly draw from a 
population will yield statistics that are plus or minus two standard errors from the population 
parameter. In our example, then, two standard errors would be .05 times 2 or .10 or 10%. So, 
95% of the samples we could draw from the population would produce estimates within 31% 
(calculated from 41% — 10%) and 51% (calculated from 41% + 10%), still clustering around 
the population parameter of 41%, but now a bit more widely. 

Finally, statisticians have determined that 99.9% of the samples we could draw will yield 
statistics that are plus or minus three standard errors from the population parameter. If we 
were to apply the three standard error increments to our example, we would be at 15% (.05 
times 3). Therefore, all but one in a thousand (99.9%) of the samples we could draw from the 
population in our illustration would produce sample statistics between 26% (41% — 15%) and 
56% (41% + 15%), now clustering decidedly less closely (i.e., dispersed more widely) around 
the population parameter of 41%. 

We hope you have noticed in the preceding paragraphs that the odds of obtaining a sam- 
ple statistic that will cluster closely around the population parameter are smaller, whereas the 
odds of obtaining a sample statistic that is dispersed more widely away from the population 
parameter are greater. In other words, we can be more confident that our sample estimate is 
within a wider range of values surrounding the population parameter compared to a narrower 
range. This is the final piece of the puzzle in sampling logic, and it introduces the key terms 
confidence level and confidence interval, as Earl Babbie (2010, p. 206) explains: 


Whereas probability theory specifies that 68% of that fictitious large number of samples 
would produce estimates failing within one standard error of the parameter, we can 
turn the logic around and infer that any single random sample estimate has a 68 percent 
chance of falling within that range. . . . We express the accuracy of our sample statistics 
in terms of a level of confidence that the statistics fall within a specified interval from the 
parameter. For example, we may say that we are 95 percent confident that our sample 
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statistics . . . are within plus or minus five percentage points of the population parameter. 
As the confidence interval is expanded for a given statistic, our confidence increases. For 
example, we may say we are 99.9 percent confident that our statistic falls within three 
standard errors of the true value. 


This discussion might remind you of the margin of error, as reported in public opinion 
polls. Indeed, the margin of error is based on confidence intervals. The Pew Research Center 
reports on its American Trends Panel data, for instance, that the data they display from U.S. 
adults who are categorized as Baby Boomers is derived from a sample size of 3,983 respond- 
ents and carries a margin of error of 2.6 percentage points (Parker, Graf, & Igielnik, 2019a). 
A poll found that 37% of Baby Boomers approve of NFL players kneeling during the national 
anthem at sporting events (Parker, Graf, & Igielnik, 2019b). We can assume at the 68% confi- 
dence level that the population parameter (if we were able to poll every single Baby Boomer) 
is somewhere between 34.4% and 39.6% who approve of this practice (calculated by subtract- 
ing and then adding 2.6% to 37%, respectively). We can assume at the 95% confidence level 
that the population parameter is somewhere between 31.8% and 42.2% (subtracting and then 
adding 5.2%—2.6% times 2—to 37%, respectively). Interestingly, in the same poll, 61% of 
those in the Gen Z generation and 62% of Millennials said they approved of football players 
taking a knee during the anthem. 


Determining Sample Size 


Although we have established the general rule that bigger is better in terms of sample size, 
there is also the issue of diminishing returns. Not only can very large samples be expensive 
and/or incredibly difficult to attain, but mathematically, the advantage of increased accu- 
racy in generalizing to the population associated with ever-expanding sample sizes becomes 
smaller and smaller to the point that it will eventually be negligible. This is why, for instance, 
national polls in the U.S. need only consist of 1,500 or fewer randomly selected respondents 
to represent certain age groups (like Millennials or Baby Boomers) even when their popula- 
tion is in the tens of millions. Mathematically, any advantage to sampling beyond those num- 
bers (for instance to 3,000 respondents or more) for generalizability is very small and therefore 
unlikely to make a difference in the phenomenon studied. 

Yet, once again, this is a general mathematical principle that will likely have little bearing 
on how you will sample in your own research. Let us get to the more practical matter of how 
you will go about determining the size of the sample to use in your own study. There are a 
number of guidelines to follow to figure out a good sample size, and those guidelines range 
from being straightforward to a little more involved. One very rough and easy guideline that 
might be especially useful for the beginning researcher is to look up published studies on the 
same topic or on a similar topic as the study you intend to run and determine what size sample 
those researchers used. Yours should be at least as large, if you can manage it! 

More formally, the standard error calculation we discussed can also be used to determine 
optimal sample size. Rather than solving for standard error in the equation, you would 
substitute in an error estimate that you are willing to live with in the study design and solve for 
the sample size. Most commonly, researchers use the 95% confidence level to do so and use a 
.05 standard error in the equation. There are a number of free, online sample size calculators 
that you can use to do this math for you. The survey company Qualtrics has a very simple 
calculator (www.qualtrics.com/blog/calculating-sample-size/) that asks for confidence level, 
population size, and margin of error. If you do not know the population size, you can estimate 
it, because as we suggested earlier in this section, once a population is large (say, in the tens 
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of thousands or larger), the math does not change that much. This online calculator shows 
that at the 95% confidence level, with 5% error accepted, and population estimated at 10,000, 
the recommended sample size is 370. If you keep the rest of the settings the same and change 
the population estimate to 20,000, a sample size of 377 is recommended. At a population 
of 30,000, 380 is the recommended number. At a population of 40,000, the sample recom- 
mendation changes to just 381. At 50,000, it is 382. You can see the concept of diminishing 
returns at play here, and you can also see that not knowing the population size precisely is 
acceptable since the sample size needed to generalize is not likely to change that much when 
the population is large. 

Another way to determine sample size is to use a general rule of thumb. One such rule 
applies to studies that test differences between or among groups, as is usually the case in 
an experiment (the subject of Chapter 6). When making comparisons between or among 
groups, a useful first step is determining the number of cells that will be created by the 
research design. The cells are the different groups that will be compared in the study. Then, 
statisticians have determined that, as a rule of thumb, you need at least 30 cases for each cell 
(Cohen, 1988). For example, imagine you are testing whether people who are shown a public 
service announcement (PSA) about wearing masks to prevent the spread of COVID-19 are 
more likely to report being willing to wear masks in public compared to those who are not 
shown the PSA. The cells are designated by an X here: 


Shown PSA Not shown PSA 
X X 


Using the 30 cases per cell rule of thumb, then, this research design creates two groups and 
therefore you would need at least 60 research participants for this study. Imagine further that 
you want to find out whether seeing the PSA in the form of a TV commercial has the same 
result as seeing it as a print ad in a magazine. With three cells, here, the basic rule of thumb 
suggests you would need at least 90 participants for this study: 


PSA as TV commercial PSA as print ad Not shown PSA 
X X X 


To add yet another perspective, we might study whether those under the age of 65 and those 
65 or older respond differently to the PSAs. Here, the design would look like this and would 
call for at least 180 participants (30 times 6): 


PSA as TV commercial PSA as print ad Not shown PSA 
<65 65+ <65 65+ <65 65+ 
X X X X X X 


There is also a more precise and sophisticated option for determining an ideal sample size 
prior to conducting a study, as well, and that is called doing a power analysis. Statistical 
power is the ability of any given statistical test to detect population parameters of particu- 
lar sizes and strengths with a specified degree of confidence. “If a study is powerful, then 
a researcher will be able to correctly find even small differences and relationships, but if a 
study is not powerful a researcher may miss small differences and relationships” (Wrench 
et al., 2008, p. 305). A power analysis can be done before a study is conducted to calculate 
the smallest sample size that can be used to successfully pick up on statistical relationships 
and differences of certain sizes. This is an a priori power analysis (called that because it is done 
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before the study is conducted), and it differs from a simple sample size calculator such as in 
the earlier example in that it accounts for the type of statistical test that will be run. We will 
introduce some of the main statistical tests used in Communication research in Chapter 8 
in our book. But until then, suffice it to say that researchers at Dusseldorf University have 
provided free, open-source software that you can use for power analysis called G*Power, 
www.gpower.hhu.de/. 


How Do You Actually Use This Information? The Realities of 
Research 


One of our pet peeves about some research methods books is that they present the ideal way 
of conducting research—like drawing probability samples—in such a manner that it makes 
the reader feel like a failure if they do not have the means to reach the ideal. It is important 
to learn the principles behind ideal scenarios in research and to understand why they are so 
optimal. Yet, real research is complicated, and it is often quite constrained by budget, time, 
and other practicalities. If you want to try to challenge the status quo and work toward social 
justice through quantitative social science research, we want you to be inspired to do so rather 
than discouraged when you realize that concessions often have to be made in the realities of 
conducting research. 

We would like to assure you that if you find yourself unable to draw a probability sample for 
the research you would like to conduct, you are certainly not alone. Joseph Erba and colleagues 
conducted a study of the research that was published between 2000 and 2014 that used survey 
or experimental design in six major mass communication journals (Erba, Ternes, Bobkowski, 
Logan, & Liu, 2018). Among the over 1,100 studies that they examined, Erba and colleagues 
found 82.6% used nonprobability samples. Just about half (51.1%, to be precise) of the articles 
used college students for their sample, a convenient source of participants for many researchers. 

We can look at the sampling approach used by Erba and colleagues themselves in their study 
of samples used in mass communication research. They chose studies published within the six 
particular journals in their sample due to the impact factor of the journal (a metric that shows 
how often articles from that journal are cited in other research) and the journal’ affiliation with 
major professional organizations in Communication. Then, any article meeting their criteria 
(quantitative, mass communication research with human participants conducted in the United 
States) was included in their analysis. Therefore, they used the logic of purposive sampling (to 
identify the journals) and then drew a census, examining every article that met their param- 
eters. Their data reflect, then, the research published within this timeframe within this set of 
journals. Yet, their data cannot be generalized to all published mass communication research. 

Of course, the Erba and colleagues (2018) study only reflects one subfield of the Com- 
munication discipline, mass communication. Nonetheless, it illustrates an important point we 
are making here: Many times it is simply not possible to obtain a probability sample, and that 
does not mean that your study is doomed or that it does not have the potential to contribute 
importantly to the field. Although it is true that studies using nonprobability samples cannot 
generalize to the wider population, knowledge on a topic can accumulate and become quite 
convincing—even by quantitative social science standards—if multiple studies using nonprob- 
ability samples are published on the same topic and point to more or less the same general 
findings. Despair not, then, reader, if your circumstances limit you—like they do for many 
if not most Communication scholars if the Erba and colleagues’ data are any indication—to 
purposive samples or other nonprobability techniques. 

Much, if not all, of the discussion of the math and the logic behind sampling error and 
confidence levels in this chapter is hypothetical and abstract. Yet, there are a few important 
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principles to extract from the illustrations and practices we have introduced here to apply in 
your own research literacy and practice. First, once again, we can see why, exactly, larger sam- 
ple sizes are preferable to smaller sample sizes in quantitative research both mathematically and 
logically. Logically, larger samples stand a better chance of reflecting the population parameter. 
This is because mathematically, large samples will generally reduce sampling error since n or 
sample size is the denominator in the standard error equation. Given that generalizability is one 
of the hallmarks of quantitative research, reducing sampling error is an important outcome. 

Second, we should remember that the variation in the population parameter is also an 
important part of the standard error equation, and when the parameter has less variation, 
sampling error will also be reduced. Therefore, you will need a larger sample size to attempt 
to reflect the population on an item with significant variation, and a relatively smaller sample 
size will suffice if the population is more homogeneous on the issue in question. 

Third, researchers can determine—oftentimes prior to conducting a study—how much 
uncertainty they are willing to live with regarding whether the results that they obtain from 
a sample are likely to generalize to the population. When researchers obtain a result from a 
sample (a statistic), ideally, they would like to be confident that that result happened not just 
due to sampling error or some kind of chance circumstance in the sample but rather is likely 
to be operating not just in the sample but in the wider population. Confidence intervals are 
used to make these wider claims within widely accepted standards among researchers regard- 
ing how much uncertainty is acceptable. 


Representing Minoritized Groups 


We have seen that there are many virtues of using probability sampling. Yet, sometimes, a 
researcher will want to focus on a subpopulation of interest that is relatively small in size 
compared to the overall population. A probability sampling strategy in which every member 
of the population has an equal chance of being selected may not yield a sufficient number 
from the small subpopulation of interest to fully explore variables within that subpopulation. 
Therefore, some additional strategies are needed. 

Before we begin with an explanation of ways to ensure that those from smaller social 
groups have sufficient representation in samples so their voices can be heard, it is important 
to think critically about the word minority in describing social groups. The word minority 
connotes two things at once: a group that is small in number and a group that faces discrimi- 
nation, oppression, or otherwise has less access to power in society. Yet, some social group 
populations are not a numerical minority but still have less access to power or representation 
compared to other social group populations. Some U.S. cities have majority Latinx popula- 
tions, for instance, meaning that over 50% of the population is Latinx. But the members of 
those Latinx populations may still not have equal power and/or may face discrimination and 
oppressive conditions more often compared to other populations. It is often useful to use the 
word minoritized rather than minority or non-majority, then, to acknowledge that discrimination 
and oppression do not always go hand-in-hand with the size of the social group. 

Researchers studying topics of concern for social justice may want to give minoritized 
groups more voice in their studies. By its very nature, taking a random sample is likely to 
result in minoritized groups that are relatively small in size in the population to also be small in 
number in the sample. In these cases, researchers may consider oversampling and/or weight- 
ing. Doing so has the potential to give voice to smaller and sometimes overlooked groups. 

Oversampling is “the practice of selecting respondents so that some groups make up a 
larger share of the survey sample than they do in the population” (Mercer, 2016). This is espe- 
cially useful to study small, underrepresented, or minoritized groups. At the time that Andrew 
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Mercer was writing for Pew Research Center in 2016, a random sample of U.S. residents that 
mirrored the national population would result in 13.4% of that sample identifying as Black 
or African American. In a sample of 1,000, then, that would equate to 134 Black or African 
American participants. That number may not be sufficiently large if your research design calls 
for exploring variables within the African American population. If you have research ques- 
tions about what Black or African Americans think (or feel or do), you will want to have 
more than just 13.4% if the sample identifies as Black or African American. Therefore, you 
would likely decide to oversample Black or African American individuals to ensure their 
larger presence in the sample. 

Weighting is the process of adjusting values in the data so that the data from particular 
groups can count as more or as less than those data would otherwise. Ifa group is underrep- 
resented in the sample compared to the population, data from members of that group can be 
weighted more heavily than data from members of groups that are not underrepresented to 
ensure that the smaller group does not get overshadowed by the larger groups. If a group is 
overrepresented in the sample, data from members of that group can be weighted less heav- 
ily than data from members of groups that are not overrepresented. Thus, some data can be 
adjusted by the researcher to carry more or less weight compared to other data. 

Weighting is typically accomplished by multiplying the data from the over- or under- 
represented group by a certain value. For instance, if you want to weight the responses from 
indigenous peoples in your sample, you can multiply all of the responses from members of 
this group by two to give those responses twice as much weight in the data set. Weighting can 
be used in conjunction with oversampling. Data from a group that has been oversampled can 
remain unadjusted when looking within that group. Yet, when the researcher wants to look 
across groups, the oversampled group’s data can be weighted downward so that the group’s 
relative size in the overall population is accurately estimated. This practice is illustrated in the 
following Research in Action segment. 


Research in Action 


The Use of Oversampling and Weighting in Public Opinion 
Research 


In this chart, we see how pollsters from the Pew Research Center oversampled Hispanic 
or Latinx respondents for a survey in 2016 such that, when data from those respond- 
ents were unweighted, they represented 24% of the sample. Oversampling allowed for 
a sufficient number of Hispanic or Latinx respondents to focus on the opinions and 
experiences of that group in analyses pertaining only to those within the group. Yet, 
in order to report in other analyses on Hispanic or Latinx respondents alongside those 
from other racial and ethnic groups, the researchers had to adjust the data from the 
Hispanic or Latinx group (weighting it down) so that the weight of that group’s data 
approximated the size of the group in the U.S. population. 
Andrew Mercer (2016) from the Pew Research Center explains: 


“A good example is a Pew Research Center survey from June of this year, in which we wanted 
to focus in depth on the U.S. Hispanic population. In the previous survey from March, 
there were 291 Hispanic respondents out of 2,254 total respondents, or 13% of the sample 
before weighting. This is pretty close to the true Hispanic share of the population (15%), 
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Note: The dashed line represents the share of U.S. adults who 
identify as Hispanic or Latino in the Census Bureau’s 2014 
American Community Survey. 


Source: Survey conducted June 15-26, 2016, and Pew Research 
Center tabulations of U.S. Census Bureau population estimates 
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Figure 4.11 Weighted sampling 


Source: ““Oversampling is used to study small groups, not bias poll results.” Pew Research Center, Washing- 
ton, D.C. (October 25, 2016) www.pewresearch.org/fact-tank/2016/10/25/oversampling-is-used-to-study- 
small-groups-not-bias-poll-results/ 


but we wanted to have more than 291 people responding so we could do a more in-depth 
analysis. In order to have a larger sample of Hispanics in June, we surveyed 543 Hispanics 
out of 2,245 total respondents, or 24% of the unweighted sample. This gave us a much larger 
sample to analyze and made the estimates for Hispanics more precise. If we just stopped here, 
estimates for the total population would overrepresent Hispanics. Instead, we weight them 
back down so that when we look at the whole sample, the share of Hispanics falls back in 
line with their actual share of the population. This way, we still have more precise estimates 
when looking at Hispanics specifically, but we also have the correct distribution when looking 
at the sample as a whole.” 
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Many Communication researchers use oversampling and/or weighting to study small and/ 
or minoritized groups. Kelly D. Blake and colleagues (Blake et al., 2016), for example, describe 
the use of the Health Information National Trends Survey to study knowledge, attitudes, and 
behavior related to cancer from a health communication perspective. For the national survey, 
geographic areas with known smoking rates that were moderate to high were oversampled 
“in an attempt to increase the yield of current smokers” (p. 1271) among those in the sample. 
An example in the context of political communication research is a study by Shanto Iyengar 
and Sean J. Westwood (2015) in which Black survey respondents were oversampled from the 
Survey Sampling International panel in order to study negative views toward members of 
an opposing political party among both Black and White research participants. In another 
example, Neal Thurman and Nic Newman published a study in Journalism Studies (2014) 
that examined news consumption in multiple countries using an online questionnaire. The 
researchers report that “The data were weighted to targets set on age and gender, region, 
newspaper readership, and social grade, to reflect the total population of each of the nine 
countries surveyed” (p. 659). 

Because the focus of this book is on conducting sound quantitative social science research 
with a commitment to social justice, oversampling and/or weighting are quite useful 
approaches. Oversampling and weighting can center the views, responses, and experiences of 
minoritized groups that have traditionally been underrepresented in social science research. 
These techniques enable the researcher to obtain a sufficiently large number of members of 
social groups that are small in the overall population and/or difficult to access for research. 

Indeed, it is crucial that we extend our research beyond groups that are large in size and/ 
or have greater access to power (such as White or heterosexual individuals) so that we can 
build knowledge that reflects the full diversity of populations of interest. The discipline of 
psychology, in which much of the knowledge produced stems from the use of undergraduate 
college students as research participants, has been critiqued for being “WEIRD,” an acronym 
that suggests it relies too heavily on Western, educated research participants from industrial, 
rich, democratic countries (Henrich, Heine, & Norenzayan, 2010). Afifi and Cornejo (2020) 
identify similar biases in the samples used to study interpersonal communication, finding 
that samples tend to overrepresent White and Asian voices, as well as voices from the United 
States, and to underrepresent Black and Latinx voices in addition to research participants 
from other parts of the world. Although Afifi and Cornejo limited their study to interper- 
sonal communication, the patterns they identify are likely to be operating in other parts of 
the Communication field, as well. We hope the content of this chapter will guide you in 
not just how you will take a sample in your research but also in your consideration of whose 
voices will be included or excluded, amplified or diminished in the sampling decisions that 
you make. 

In addition to reflecting on whose voices will be foregrounded in your research, another 
important consideration pertaining to sampling is knowing and respecting the norms and 
values within a group to be studied. In some research contexts, utilizing a random sample 
in order to achieve the principle of representativeness can fly in the face of local norms and 
values. If researchers select participants for their studies randomly or by some systematic pro- 
cedure, this may result in those not selected feeling slighted or pushed further to the margins. 

Mohatt and Thomas (2006), for instance, have found that random sampling is not the right 
choice when studying some indigenous communities. They explain: 


Random sampling procedures violate a fundamental principle of every indigenous group 
with whom I have worked. It assumes that a statistical or mathematical rationale should 
determine whom we talk to or with whom we intervene. It is . . . both exclusive and 
dangerous because not all members of the community would be included, and there 
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would be no evidence of co-membership on the part of the researchers and therefore no 
sense of protection from harm. 


(pp. 110-111) 


If the practice of random selection in sampling violates the local norms of the community 
to be studied, a different approach is needed. Purposive or snowball sampling techniques 
might be the better option in a scenario such as this, tipping the scales in favor of respecting 
the values of those being studied and thereby gaining their trust and participation rather than 
upholding the typical quantitative practice of prioritizing probability sampling. For a more 
detailed account of conducting quantitative research in and with indigenous populations, we 
encourage you to read Indigenous Statistics: A Quantitative Research Methodology (2013) by Mag- 
gie Walter and Chris Andersen. 


Summary 


We have learned a number of approaches to comprising a research sample and considered the 
basic math behind our degree of confidence that a randomly drawn sample can do a reason- 
able job of representing the population. In doing so, we have introduced probability sampling 
techniques as well as nonprobability-based approaches to sampling and learned why the for- 
mer are generally superior if they are feasible. We have seen that representative samples and 
generalizable findings are typically the goal in quantitative research methods, constituting a 
significant part of why there is the potential to harness the “power of numbers.” 

In this chapter, we have shown some ways to determine what size samples you will want to 
try to achieve in your research studies, and we have introduced the basic principles behind those 
formulas. We have illustrated the value of oversampling and the utility of weighting in sampling, 
particularly for their ability to allow for minoritized groups to be represented in quantitative social 
science research. We offer insights on how to be respectful of the norms of groups and communi- 
ties accessed for research purposes. This discussion of sampling sets the stage for the next chapters, 
which will provide an in-depth look at surveys, experiments, and content analyses, all research 
methods in which the researcher will typically take a sample from a larger population. 


Discussion Questions and Activities 


1. What are the pros and cons of the different sampling techniques—both probability and 
nonprobability types—that were introduced in this chapter? 

2. What are some obstacles or challenges associated with sampling? 

3. Why are larger samples typically better in quantitative research? What is the logic behind 
the use of a fairly large sample? 

4. Imagine you are interested in doing a study of the comments that are posted to a New York 
Times online edition article on immigration. How would you approach sampling for this 
topic? 

5. Now imagine that you would like to know whether watching a Sesame Street episode fea- 
turing Julia, a character who has autism, leads to greater understanding of autism among 
preschoolers. How would you approach sampling for this particular study? 
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5 Soliciting Opinions Through 
Survey Research 


News stories that we read in our everyday lives often refer to survey research findings, includ- 
ing those that show patterns pertaining specifically to social justice issues. For example, survey 
research studies show that seven in ten teens in the U.S. see anxiety and depression as a seri- 
ous problem in their age group (Horowitz & Graf, 2019). Fifty percent of respondents in a 
recent survey in the U.S. say that men getting away with committing sexual harassment and 
assault is a major problem in the workplace (Graf, 2018). In terms of topics such as the digital 
divide and access to technologies, survey studies have revealed large discrepancies around the 
world. For instance, whereas countries like South Korea and the Netherlands have more than 
90% adoption rates for mobile phones, the rate is closer to four-in-ten in emerging econo- 
mies such as Indonesia and Nigeria, with India only reporting 24% smartphone penetration 
(Taylor & Silver, 2019). All of these interesting insights on social justice issues would not have 
been possible without survey research. 

Surveys are one of the most popular research methods used by social scientists. They involve 
collecting data from a sample of participants based on their responses to items posed to them 
on a questionnaire. A survey can be defined as “a social scientific method for gathering 
quantifiable information about a specific group of people by asking the group members ques- 
tions about their individual attitudes, values, beliefs, behaviors, knowledge, and perceptions” 
(Wrench, Thomas-Maddox, Richmond, & McCroskey, 2008, p. 214). Surveys are typically 
self-reported, meaning that the respondents read the items and answer them on their own, 
without assistance from a researcher or interviewer. However, some types of survey research 
could also involve interviewers who pose the questions to the study participants. 

Nearly every topic in Communication has been studied using survey methods. Surveys 
are used across a broad spectrum of organizations ranging from those in academia to govern- 
ment agencies to marketing firms. Even if you have never conducted your own survey study, 
chances are that you have participated in one at some point in time. News articles routinely 
refer to survey studies in providing context and background to topics that they are covering. 
Be it political pundits explaining voting patterns in elections, marketing gurus selling products 
and services, nonprofit volunteers evaluating the effectiveness of a program, community lead- 
ers assessing local needs, or government officers making policy decisions, surveys are used in 
a variety of settings. 

By definition surveys give voice to participants through collecting their opinions, expe- 
riences, reports of behavior, and other measures. Therefore, they are both a popular and a 
powerful tool for social justice-related concerns. In this chapter, you will learn about the pros 
and cons of surveys, when and how to use surveys for social justice research, the basics of 
survey design and implementation, and ways to reduce common errors. You will learn how to 
become more discerning readers of survey studies by evaluating surveys conducted by others, 
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as well as how to design your own survey in a culturally appropriate manner to address social 
problems or issues of interest to you. 


Advantages and Disadvantages of Surveys 


Surveys are easy, efficient, and versatile methods used across a broad range of social settings 
and communication contexts (Schutt, 2015). The advantages of surveys are their relative 
ease and cost-effectiveness of data collection. They are efficient because data can be collected 
rather swiftly by administering surveys to a large sample in one go. Surveys allow for substan- 
tial amounts of information to be collected, often without expending a great deal of money 
and time. They can be administered through different types of modes: mail, telephone, or 
online. Mailed surveys often involve several pages of questions, which can help collect a lot of 
valuable information in one relatively inexpensive study. Phone surveys and in-person surveys 
can last for several minutes, which also allows for a great deal of data to be gathered at once. 
The advent of technologies such as mobile phones, tablets, and laptops makes it even more 
convenient and confidential to conduct surveys. 

Surveys are the method of choice for quantitative researchers when the central purpose 
of a research project is to paint a representative picture of a broad range of characteristics 
of a population. In general, surveys offer high external validity and generalizability of 
findings from a sample to the population at large. Just as we learned in Chapter 4, census 
studies refer to those surveys where the entire population is included, rather than a sample 
of participants. While census studies are ideal, they can be expensive and impractical in 
most situations. Therefore, samples are quite frequently used. As explained in detail in 
Chapter 4, probability sampling allows for greater generalizability of findings compared to 
nonprobability sampling. Indeed, surveys are a great choice for using probability sampling 
techniques. In general, the representativeness and size of the sample used for a survey 
determine its quality. 

Survey data can help establish correlations, which are measures of mutual relationships 
between two or more variables. A correlation is a statistical measure that tells us the extent to 
which two variables are mutually interconnected. For instance, a positive correlation between 
two variables means that as the values of one variable increases, the other one also increases. 
For example, survey findings could reveal a positive correlation between trust in science and 
mask-wearing to prevent the spread of the coronavirus. That is, the greater the faith in sci- 
ence, the more likely people are to wear a mask. Conversely, the lesser the faith in science, 
the less likely people are to wear a mask. Or there could be a negative correlation between 
variables, as well, such that as one variable increases, the other decreases. For example, con- 
servatism and mask-wearing could be correlated such that those who report higher scores on 
conservatism indicate that they wear masks less often. We will learn more about correlations 
in subsequent chapters. 

Like all research methods, however, surveys are not without disadvantages. Although sur- 
veys can help determine correlations between variables, they cannot establish causation. We 
will learn more about causal relationships in the next chapter on experiments, but it is impor- 
tant to recognize that one disadvantage of surveys is that it is hard to establish the time order 
of variables. In other words, even if we know that two variables are correlated and mutually 
associated, we do not know for sure whether one is the cause of the other or whether they 
merely co-exist without having a cause-effect relationship. Other disadvantages of surveys are 
that in-person and phone surveys can take time, effort, and money when interviewers have to 
make several follow-ups to reach the respondents. We would be willing to bet, for instance, 
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that when your own phone rings, and it is someone asking you to complete a survey, you 
may not be inclined to pick up the phone or devote the time necessary to respond. These 
costs could range anywhere from a few dozen dollars to several hundreds of dollars when they 
involve repeat visits and traveling distances. Although electronic surveys conducted via email 
or websites are much faster, response rates are often not good. Web-based surveys also leave 
out those without reliable access to the internet, such as households with lower income and 
those with elderly residents. 


Role of Surveys in Social Justice Research 


A well-designed survey for social justice purposes seeks to find relevant information, allows 
respondents to share their experiences honestly in a confidential manner, and uses the find- 
ings to lead to actionable outcomes that serve the respondents and their community. It is 
important to place participants’ best interests at the center at every stage of survey design, 
especially when the goal is social justice-oriented. Answering surveys takes time and effort on 
the part of the respondents. Participants, especially those from under-resourced groups, have 
no obligation to complete surveys, especially if researchers do not attempt to make it relevant 
to them or meaningful for them. 

Surveys help those committed to social justice concerns to understand the complex realities 
of social issues that lead to injustices and inequalities. They can help identify the key variables 
where energies need to be focused to bring about transformation and dismantle existing ine- 
qualities and injustices. Within Communication, many social justice topics have been studied 
using surveys: health disparities, civic engagement, political participation, digital inequali- 
ties, food insecurity, racial discrimination, sexual harassment, anti-discrimination, and labor 
organizing are some examples. For example, within organizational communication contexts, 
survey methods have been used in understanding toxic waste storage in nuclear power plants 
(Barbour & Gill, 2014), in understanding how a game can increase STEM-field participation 
(Barbour, Ballard, Barge, & Gill, 2017), and in organizing around the Occupy Wall Street 
movement (Costanza-Chock, 2018b). 

Since social issues often involve multiple stakeholders with different needs and motiva- 
tions, surveys can help gain valuable input from relevant groups to enhance understanding of 
the issue in an efficient and relatively quick manner. They allow researchers to examine the 
patterns of relationships across subgroups within a population and lend themselves well for 
cross-cultural and cross-national comparative analyses of attitudes, opinions, behaviors, and 
intentions. Importantly for social justice research, surveys can be used to amplify the voices 
of minoritized groups and/or shed light on social issues or conditions that are experienced 
within and across diverse populations. For example, they help compare how different genders 
might relate to the #MeToo movement, how various racial groups experience police brutal- 
ity, how refugees in different countries stay in touch with family through technology, or how 
cyberbullying differs across age groups. In other words, surveys are helpful in establishing 
cross-population generalizability by comparing local conditions within one subgroup with 
those of another cultural, social, or other types of subgroups. 

Surveys offer the opportunity to collect sensitive information about topics such as dis- 
crimination, abuse, violence, or inequities while giving the respondents a safe space to express 
themselves without fear of retaliation, embarrassment, or other negative feelings that come 
from the disclosure of personal information. It is crucial to protect the interests and identi- 
ties of the respondents, especially if they belong to vulnerable groups such as undocumented 
immigrants, LGBTQ+ individuals, and those with disabilities who might wish to express 
themselves without disclosing their identities. Anonymity refers to the situation where 
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nobody, including the researcher, knows the participant’s identity. Whenever possible, it is 
best not to collect any identifiable information by keeping the survey anonymous. If anonym- 
ity is not possible to assure (perhaps for reasons we discussed in Chapter 2 about Institutional 
Review Board protocols), then researchers should at least make sure that the information 
collected from participants is kept confidential. Confidentiality means that the researcher 
knows the identity of the participant, but they take care not to reveal it to anyone else. When 
conducting surveys over a long period of time with the same population, it is especially 
important to maintain confidentiality, as identifying information is needed to contact the 
respondents over various stages of the study. 

In general, respondents weigh the potential costs and benefits of survey participation before 
determining whether to accept or decline an invitation to participate in such research. If 
respondents do not see the value in participating in the survey, they are unlikely to be per- 
suaded to invest their time and effort in it. If the survey is on a topic that they care about, are 
interested in, and see as relevant to themselves, they are more likely to participate. Surveys 
that are easy-to-understand, accessible, and attractive are also more likely to elicit responses. 
Additionally, if respondents believe that their input on the surveys will be taken seriously 
and may even influence decisions, they are much more likely to participate. This last point is 
especially true for social justice issues, where the potential for actionable outcomes that could 
reduce social inequalities and alleviate difficulties in everyday lived experiences could moti- 
vate participants to take part in the survey. 


Cross-Cultural Survey Research 


When conducting cross-cultural survey studies, it is important to understand how dif- 
ferent cultural groups might respond to the same items differently, even if their experiences 
are the same. They could also vary greatly in their willingness and openness in responding to 
researchers who are seen as “outsiders.” For example, in certain subgroups, women partici- 
pants might see men who are collecting survey responses as outsiders to whom they do not 
feel comfortable responding, leading to nonresponse. At the same time, it is also possible that 
some respondents might especially want to give socially desirable answers that they think are 
expected by the researchers out of politeness, especially when there is a large status or class 
difference between the researchers and respondents. Social desirability is a key limitation to 
survey research that we will return to later in this chapter. 

Various groups could differ greatly in their access to infrastructure, including access to the 
internet, mobile phones, and computers. Some of the groups of interest might not be well- 
connected by transportation or communication channels. Some people simply might not own 
a smartphone, a computer, or have easy access to the internet. Even if they do have access, 
respondents could vary greatly in their levels of digital literacy, which may influence the ease 
with which they navigate online surveys, for instance. Depending on which populations the 
researcher is studying, they must pick an appropriate method of administering their surveys so 
it is accessible to most of the participants. 

Although such events are fairly rare, there could also be political barriers that influ- 
ence survey responses when conducting multinational cross-cultural surveys. Some countries 
such as North Korea ban survey research, making it impossible to collect data there. In some 
countries, certain topics might be explicitly restricted from being included in survey research, 
whereas in other countries, even if there are no legal restrictions, the social norms and customs 
might make data collection infeasible. Internet censorship is also an issue in some countries 
where researchers might have to take additional steps to ensure the safety and confidentiality 
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of the participants. As discussed in our chapter on ethics, the safety and well-being of the 
researchers and participants should be of foremost concern when making such decisions. 
Language is an important consideration when conducting cross-cultural and cross-national 
survey studies. In several geographic regions around the world, respondents are bilingual and 
multilingual. Even within a single nation or region, one cannot assume cultural or linguistic 
homogeneity. Respondents’ level of fluency and literacy could be different across various lan- 
guages they know. Languages that they understand, read, write, and speak might not all be the 
same ones. As social justice scholars, in particular, it is more likely that the sample that is being 
studied consists of respondents who might not be fluent in the dominant language used by the 
majority groups. It is even possible that some languages do not have written scripts, in which case 
only face-to-face or phone surveys are possible. In countries where immigrants make up a high 
percentage of the population, it is likely that they are more conversant in a language different 
from the majority native population. Therefore, it becomes crucial for social justice scholars to 
be sensitive about making the surveys available and accessible in the language of the respondents. 
Well-translated questionnaires are equivalent in conveying the meanings intended in the 
original text of the survey. They should be consistent and reliable so that they are similar to the 
original survey in meaning, style, and tone. Rather than designing a survey in one language 
first and then translating into other languages, it is best to involve multiple language versions 
of the survey from the beginning stages of the survey design. It is helpful if the translation is 
done not just by one person but by a team of researchers who have expertise in questionnaire 
design, the cultural context, and language. It is crucial to review, pretest, and back translate 


Figure 5.1 Different cultural groups might respond to the same items differently, so it is important for survey 
design to be culturally sensitive 


Source: (U.S. Navy photo by HMC Josh Ives/released) www.flickr.com/photos/isafmedia/8465011090/ 
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the questionnaire to check validity and reliability. Back translation is the method used by 
researchers to translate a questionnaire back into the original source language to ensure there 
are no errors. For example, the second author of this book was part of a research study con- 
ducted in South Korea (with 352 respondents) and India (with 333 respondents) about how 
American television consumption is correlated with feelings of relative economic deprivation 
(Yang, Ramasubramanian, & Oliver, 2008). In this study, participants completed surveys in 
the Korean language and English. The researchers used the back translation method to make 
sure that there were no errors in the translations. 

Before conducting surveys with the target population, it is helpful for researchers to spend 
some time as researchers to understand the cultural norms, social expectations, and other 
constraints of the participants. Such input shapes survey design at every stage, from the topics 
being examined to the modes of data collection. For instance, in a community-driven project 
on stigma management among gender and sexual minorities in a subaltern community of 
South India, an advisory board guided the theory-building and research design processes for 
a research project based on their insights about everyday lived stigma experiences (Thaker, 
Dutta, Nair, & Rao, 2018). The surveys were translated into the local Telugu language from 
English and then were back translated to check for consistency. They were pretested in the 
field to make sure questions were properly understood and relevant. 


Omnibus Surveys and Publicly Accessible Datasets 


In addition to collecting data firsthand, survey researchers have two additional options. They 
can hire a research firm for collecting data, or they can use survey data that already 
exists. Several research firms are oriented toward academic and social research and can pro- 
vide access to participants for survey researchers for a cost. Some examples of research firms 
are Qualtrics, Survey Monkey, Question Pro, Dynata (formerly Research Now), Prolific, 
and Knowledge Networks. Researchers can pay these organizations to field a survey study 
for them, providing a certain number of respondents to complete the questionnaire. Another 
option for researchers looking to find respondents willing to fill out questionnaires is to use 
Amazon’s Mechanical Turk (MTurk) platform in which participants are compensated for 
small research tasks. There are also independent organizations such as the National Opin- 
ion Research Center (NORC) that host large-scale surveys. NORC provides access to data 
generated from a vast number of surveys, many of which use nationally representative sam- 
ples in the United States. Many of these large-scale omnibus surveys have the option for 
researchers to add a few questions of interest to them. An omnibus survey is one that covers a 
wide range of topics from multiple researchers. These are wonderful opportunities to obtain 
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Figure 5.2 Language is an important consideration when conducting cross-cultural surveys 


Source: Wikipedia user Jn0101, https://commons.wikimedia.org/wiki/File:Devanagari_keyboard.png 
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broad and far-reaching data, and yet these opportunities often come at a financial cost to the 
researcher. 

Not all researchers will have the time and money necessary to conduct or purchase survey 
data from large-scale survey research. If you are a researcher interested in social justice issues 
but do not have the resources to conduct your own surveys across time or across various 
subgroups of interest to you, another option is to use publicly available datasets. There are 
several community-oriented organizations that conduct surveys on key social issues. Many of 
them have the resources and funding to collect data across many geographic areas and over a 
long period of time. The following table is a compilation of some of the organizations whose 
survey research reports could be of use to those who study social issues. 


Some Publicly Available Datasets for Social Justice Researchers 


1. American National Election Studies: These nationally open surveys are often 
conducted before and after U.S. presidential elections on topics such as partisan- 
ship, candidate evaluations, social issues (such as welfare, race policies, economic 
issues, foreign relations, environment, and federal spending), values (such as reli- 
gious values, moral orientation, social trust, and altruism), political participation, 
civic engagement, and media use. 

2. Data.gov: Hosted by the U.S. General Services Administration, this site provides 
data on topics such as agriculture, climate, health, education, and public safety col- 
lected from across various federal government agencies. 

3. General Social Survey (GSS): Established in 1972, this is a probabilistic survey 
based on personal interview data monitors changes in social characteristics and 
attitudes in the U.S. It examines topics such as demographics, behavioral and atti- 
tudinal data on crime, violence, civil liberties, tolerance, morality, stress, national 
spending, and trauma. 

4. Health Information National Trends Survey (HINTS): This resource features 
national surveys on cancer-related information by the National Cancer Institute. 
It also provides information on a range of other topics such as health information, 
communication patterns relating to health, information on cancer risks, food and 
medical products, health knowledge and perceptions, physical activity, health lit- 
eracy, and health status. 

5. Inter-University Consortium for Political and Social Research: Housed 
in the University of Michigan, this center has compiled over 10,000 datasets from 
around the world on topics such as education, mental health, immigration, happi- 
ness, domestic violence, depression, alcohol and substance abuse, religion, obesity, 
and crime. Some of the most downloaded datasets are those on the Chinese Lon- 
gitudinal Health and Longevity Survey, National Longitudinal Study of Adoles- 
cent to Adult Health, Indian Human Development Survey, and Maternal Lifestyle 
Study. 

6. Kinsey Institute for Research in Sex, Gender, and Reproduction: This 
resource has archival data based on surveys relating to sex, sexuality, gender, and 
reproduction, including data on LGBTQ+ individuals. 

7. Minority Data Resource Center (MDRC): The Resource Center for Minor- 
ity Data provides expansive and in-depth data on racial and ethnic minorities in the 
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US. It archives and assembles existing datasets focused on communities of color. 
Some topics covered include education, employment, discrimination, public opin- 
ion, political participation, health and well-being, history, poverty, and income. 

8. National Center for Health Statistics: This organization offers downloaded 
public use datasets through the Center for Disease Control on topics such as nutri- 
tion, vital health statistics, family growth, immunization, and aging. 

9. Pew Research Center: A nonpartisan fact tank, the Pew Research Center pro- 
vides open datasets on topics such as U.S. politics and policy, journalism, media, 
internet, science, technology, religion, public life, race, and also global trends and 
attitudes on a range of social issues. 

10. UNESCO Institute for Statistics: This group provides statistical data on a wide 
range of social indicators from around the world such as education, culture, science, 
and communication. Its mission is to use data to transform lives. Some examples 
of datasets and reports are State of the World’s Children, Global Gender Gap, ICT 
Development Index, and World Inequality Databases on Education. 


Cross-Sectional Versus Longitudinal Survey Designs 


Generally speaking, survey designs can be divided into two broad types: cross-sectional and 
longitudinal surveys (Figure 5.4). Cross-sectional surveys are those that provide a descrip- 
tive snapshot of the relationships among variables in that given moment. Participants complete 
the survey at one point in time. For example, in the second author’s study on how televised 
stereotypes of racial/ethnic minorities are correlated with views on affirmative action policies 
(Ramasubramanian, 2010), the goal was to examine the relationships among key variables 
and any differences between groups (such as high-prejudice versus low-prejudice individuals 
in the sample) at a particular point in time. 

If the researcher is interested in exploring how the relationships change with time, longi- 
tudinal survey designs would be more appropriate. Longitudinal designs involve admin- 
istering the survey more than once over a period of time. Because multiple surveys are 
administered in succession, longitudinal designs provide information on the order in which 
the variables might influence one another. For example, researchers have tracked how stress 
levels in college students change as a semester progresses using weekly surveys (Garett, Liu, & 
Young, 2017), clearly a longitudinal survey given that the data are gathered across multiple 
time periods. Another example of a longitudinal study is one conducted by the second author 
of the book using online surveys administered at a three-week interval over a nine-week 
period (Saleem, Yang, & Ramasubramanian, 2016). That study examined how reliance on 
media versus direct contact with Muslims is connected to negative attitudes and beliefs about 
this group over a period of time. 


Trend Studies 


There are many types of longitudinal survey designs including trend, cohort, and panel. In a 
trend design, the researcher is interested in seeing how trends change with time with respect 
to a topic of their interest. In trends, different groups of people are included as respondents 
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Figure 5.3 Cross-sectional vs. longitudinal design 


Source: Emily Riewestahl 


at different points in time, but the topic or topics remain the same. Several large-scale sur- 
veys such as the American National Election Studies, the General Social Survey, and the 
Health Information National Trends Survey track many important social indicators such as 
social characteristics, religious practices, political knowledge, public opinions on policy issues, 
health information-seeking, health literacy, and opinions about other social issues such as 
equal pay for women, open housing, desegregation, etc. 


Cohort Studies 


Another type of longitudinal survey design is a cohort. A cohort study is when a cohort 
of people with a shared characteristic are tracked over a period of time. For example, if 
you are interested in tracking how the COVID-19 pandemic influences the mental health 
of graduating high schoolers, you might conduct a survey with a sample of graduating 
seniors one month, six months, and 12 months after graduating from high school in 2020. 
As you follow this cohort over time, you need not include the same exact participants each 
time you administer your survey, but you would make sure that they are sampled from 
that group of respondents who share the characteristic of having been graduating seniors 
in 2020. 


Panel Studies 


The last type of longitudinal survey design is a panel. You can think of panels as a specific 
type of cohort study where you are following the same individual members of the cohort 
throughout the length of the study. Panel studies collect multiple sets of data repeatedly from 
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the same set of individuals over time. Panels are commonly used in audience research, new 
product testing, and in medical research. They provide fine-grained individual-level data and 
can study how that data changes (or does not change) over time. However, they are expen- 
sive and, of course, time consuming to conduct. There could be attrition, which refers to 
respondents dropping out of the panels. When that happens, the size of the sample drops 
too, of course, and therefore panel studies often have to start out with a very large sample 
in anticipation of the fact that not all participants will stick with the study for its duration. 
Several of the Pew Research Center study reports mentioned at the beginning of this chapter 
were collected from their panel, which was started in 2014 with over 5,300 participants and 
has now included online surveys in addition to phone surveys (Gramlich, 2019). This panel 
has covered topics such as media consumption, personality politics, religion and technology, 
science, political knowledge, immigration, and political antipathy. 


Creating Reliable and Valid Survey Measures 


Now that we have looked at the various types of survey designs, we will now move into the 
nitty gritty details of how exactly to design a survey questionnaire. A good survey question- 
naire is one that yields reliable and valid answers. Clear, interesting, and well-organized items 
increase the reliability and validity of measures. They encourage respondents to engage care- 
fully with the questionnaire and take their participation seriously. Reliability in a survey 
context refers to the extent to which the answers are consistent. The same survey has to be 
administered to all respondents. As a survey researcher, one cannot rephrase an item or elabo- 
rate on it for some respondents because that would result in a lack of consistency, which makes 
the measure unreliable. The items must also be consistently understood by all respondents. 
Items should be easy to comprehend, without any ambiguity or lack of clarity. 

According to applied researchers, Bickman and Rog (2009), “There are four basic charac- 
teristics of questions and answers that are fundamental to a good measurement process: 


=. 


Questions need to be consistently understood. 

Respondents need to have access to the information required to answer the question. 

3. The way in which respondents are asked to answer the question must provide an appro- 
priate way to report what they have to say. 

4. Respondents must be willing to provide the answers called for in the question” (p. 376). 


N 


The common errors that survey researchers must avoid are measurement error, nonresponse, 
and poor sampling. Measurement error refers to problems in the quality of surveys caused 
by not asking appropriate items in a clear and unbiased fashion. Nonresponse is when par- 
ticipants decline to participate in surveys or if they participate in the survey but skip a large 
portion of the questions on it. By making surveys more accessible and inclusive, establishing 
trust with respondents, and increasing the benefits for participation, response rates can be 
improved. Another error to avoid is poor sampling. The quality of a survey is determined 
by the size and representativeness of the sample as well as the quality of the sampling frame. 
Recall that in Chapter 4, we learned that a sampling frame is a list of those people or units 
from the population from which the sample is drawn. Ideally, the sampling frame is the same 
as the population, but it may not always be identical because of missing units. 

The best way to get useful and meaningful answers from respondents is to present the 
right items. Although this sounds simple, it requires skill and care. Presenting appropriate 
items increases response rates and reduces measurement error. Whereas increasing sample size 
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or response rates can incur financial costs, improving question quality is an inexpensive and 
efficient way to significantly improve the quality of the survey data that are collected. This 
is especially relevant when working on social justice issues because resources are often con- 
strained and funding is hard to obtain. Therefore, as a social justice researcher, it is prudent 
to invest time and energy into presenting the right items in order to yield the best error-free 
results. 


Are the Items Necessary, Unbiased, and Clear? 


Researchers should avoid confusing phrasing. For instance, instead of asking for “geo- 
graphic location,’ be direct in your approach and ask: “In which city or town do you 
live?” Items that are understood by more educated and privileged groups may not convey 
the intended meaning to marginalized groups. Avoid complicated words, jargon, and other 
expressions unfamiliar to your respondents. It is a good idea to break up complex issues into 
simpler parts and provide additional context to increase clarity. For example, when you want 
to measure sensitive issues such as discrimination or prejudice, make sure to explain what such 
phrases might mean. Instead of asking one broad item such as “Have you experienced dis- 
crimination?,” the concept of discrimination could be broken down into smaller related vari- 
ables such as “How often have you been followed around in a store because a store employee 
seemed suspicious of you?,” “How often have you been treated unfairly at work?,’ and so on. 
Sometimes a simple change of wording could save respondents a lot of mental effort. Instead 
of saying, “How many years of education have you had?,” it would be much easier to respond 
to “What is the highest degree you have received?” 

A related issue when it comes to working with specific subpopulations is avoiding 
biased items. Items should be designed to fit the knowledge level and cultural contexts 
of participants. A biased item could read something like, “Do you think this unfair policy 
about dress codes in schools should be abolished?” Perhaps some respondents do not believe 
the policy is unfair and their opinions do not line up with those of the researchers. Using 
a term such as unfair (or controversial or strongly critiqued) might provide misleading 
answers. Even when using existing measures from previously published studies, research- 
ers should review them carefully to check their suitability for the population of interest 
to them. Biases can be much subtler as well. When researchers ask, “How often do you 
exercise?” it is possible that those respondents who do not exercise at all feel uncomfortable 
answering the item honestly. 

Item clarity can also be increased by avoiding double negatives. Double negatives are 
items that have two negative phrases. For instance, an item that is worded as, “Do you disa- 
gree with legislation that would prohibit same-sex marriage?” is a sure way to confuse the 
respondents, as they have to spend a lot more energy on figuring out what, exactly, the item 
is asking. Instead, it would be better to simply have a statement such as “I support legislation 
that allows for same-sex marriage,’ (or, conversely, “I do not support legislation that allows for 
same-sex marriage”) to which participants indicate their level of agreement. 

A common mistake that even experienced survey researchers make is having double- 
barreled items on their survey. Double-barreled items are those in which one survey ques- 
tion includes two different topics but only provides one answer option. An example of such 
an item could be, “How satisfied are you with the ways in which government agencies and 
community organizations are dealing with food insecurity in your community?” Perhaps the 
respondent is satisfied with the government agencies and not community organizations or 
vice-versa. Yet, because there was only one answer option, there is no way to know what the 
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participant intended to say. The way to solve this issue, then, would be to separate the item 
here into two items. 


Are the Response Options Appropriate? 


Even if the item is clearly understood, sometimes respondents might not be able to report 
what they want to share unless they are provided with the appropriate answer options. 
The response categories should be fixed and consistent for all the respondents. As we 
learned in Chapter 3, response choices should reflect meaningful and distinctive options as 
well as cover the entire range of possible answers. Response options should be collectively 
exhaustive, which means that, together, they should offer a full range of answer options. 
For example, an item on gender might just give male and female as options, but it could be 
that the respondent identifies as a non-binary individual. When the alternatives provided do 
not reflect the spectrum of sentiments or experiences or identities, respondents might not 
answer accurately. They might also feel marginalized. Therefore, the International Com- 
munication Association changed its member profile options to reflect a greater diversity in 
gender expression by using the following options where one could check all that apply: 
cis/cisgender, genderqueer, intersex, man, non-binary, third gender, trans/transgender, 
woman, and none of the above. 

When a range of answer options is provided on items that measure attitudes, the number of 
“negative” and “positive” answer options should be balanced. For instance, when respondents 
are asked how they would classify the portrayals of Native Americans in popular films, the 
answer options could be: extremely negative, somewhat negative, neither negative nor posi- 
tive, somewhat positive, and extremely positive. In this example, we note that in a 5-point 
scale, there is a balance between the positive and negative options to choose from. 

Answer choices should also be mutually exclusive such that one and only one possible 
answer choice is used by any given respondent. For example, when measuring demographic 
variables such as income range, age, or education, make sure that the categories do not overlap 
with one another. As we have seen in Chapter 3, when measuring daily hours of exposure 
to television, for example, if the answer choices are “none, < 2 hours, 2—4 hours, 4—6 hours, 
and > 6 hours,” someone who watches four hours of TV a day would not know whether to 
fill out 2—4 hours or 4—6 hours a day. 

In Chapter 3, we learned about various levels of measurement, including Likert scales. Sur- 
vey researchers should think carefully about what level they would like their answer options 
to measure—nominal, ordinal, interval, or ratio. Likert scales are commonly used in surveys 
as answer options to measure attitudes and opinions. In a Likert scale, a statement is presented 
to the respondent, followed by a range of options indicating the level of agreement or disa- 
greement. When using Likert or Likert-type scales, make sure that the answer options are 
distinct enough from one another. For response categories ranging from “extremely satisfied” 
to “not at all satisfied,” if you use two answer options as “moderately satisfied” and “somewhat 
satisfied,” these two terms are too similar to each other to provide any meaningful distinc- 
tions. The answer options for Likert scales are typically five- or seven-point answer options. 
For example, in a survey conducted by the second author, exposure to positive Japanese 
manga heroines was correlated with wishful identification with them among American teens 
(Ramasubramanian & Kornfield, 2012). In this study, participants were asked to rate the 
extent to which they agree with statements such as “I wish I could be more like my favorite 
shojo heroine” and “Td like to do the kinds of things my favorite shojo heroine does.” The 
respondents were provided with a five-point Likert scale that ranged from 1-strongly agree to 
5- strongly disagree. 
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A related decision is whether or not to include labeled or unlabeled response options. 
In labeled response options, every point in the scale is spelled out. For instance, labeled 
responses could be: 1—not at all safe, 2-somewhat safe, 3-safe, 4—very safe, and 5—extremely 
safe. In contrast, an unlabeled version would simply label the endpoints “1—not at all safe” and 
“5—extremely safe” and indicate the other numbers in between without labels as 2, 3, and 4. 
Whenever possible, it is better to use labeled response options rather than unlabeled ones, as 
they offer more reliable answers (Schutt, 2015). 


Are Respondents Able to Answer the Items? 


Respondents can only answer those items for which they have access to information. 
If they do not know or cannot recall information, they cannot provide useful answers. For 
example, it is possible that they are simply unable to remember past events with much accu- 
racy or detail. So, they may report that they have done something when they might not have, 
and vice-versa. This could lead to a lack of consistency in responses, which can also then 
affect the validity of the survey, as it no longer measures what it seeks to measure. 

It is possible that respondents simply do not know or have access to the information that is 
being asked in the survey. For instance, individuals who are diagnosed with certain symptoms 
might be aware of their experiences but might not have been told about the specific name of their 
condition or illness. A common related error that survey researchers make is asking respondents 
to report about others’ experiences and opinions, which they do not know. An example of this is 
when certain members of the household do not have information that others might have on such 
topics as employment, health condition, nutrition, insurance, repairs, and taxes. 

There might also be situations in which the respondent does know but is not able to 
recall the relevant information. Survey methods are mostly valid for measuring situations 
where respondents are conscious of their behaviors. When the information is collected ret- 
rospectively and there is a large time lag between the experience and when the survey is 
completed, the respondents might not be able to recall. This is a common issue in measuring 
media exposure, for instance in items such as “How many minutes did you spend on your 
mobile phone yesterday?” Since mobile phone use is so ubiquitous in a world in which many 
people report that they are always online and permanently connected, respondents are not 
always able to recall how much time they spent previously with mobile phones and/or on 
social media (Naab, Karnowski, & Schliitz, 2018). 

It is best for survey researchers to consider whether they are asking respondents to recall 
answers about a reasonable duration of time. It might be easier to answer items about more 
focused time periods such as, “How many days last week did you take your prescribed 
pill?” rather than, “During the last 12 months, how many times did you take your prescribed 
pill?” It is also generally not possible or meaningful for participants to store or recall trivial 
information. Unless you are asking about major incidents, try to keep the focus of the ques- 
tion to the last week or month rather than the last year or longer. Helping your respondents 
remember specific past events is called aided recall, and this approach yields better answers. 
Instead of asking a generic item such as “How often did you help with household chores last 
week?,” it would be better to help recall by listing specific household chores and asking about 
the duration of time spent on each. 


Are Respondents Willing to Answer the Items? 


Survey respondents should feel comfortable and be willing to share their answers. Only then 
are answers valid and able to capture the lived experience that the item is trying to capture. 
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Social desirability bias refers to how respondents might give “acceptable” answers rather 
than honest ones in order to conform to societal expectations. Respondents might distort 
their responses to survey items to make themselves look good or avoid looking bad. Survey 
researchers should do their best to reduce social desirability among their respondents to gather 
meaningful responses. For example, depending on the cultural contexts, certain behaviors 
such as sexual violence, premarital sex, alcohol consumption, multiple sexual partners, abor- 
tion, drug use, etc. might be taboo topics since they might be seen as illegal, immoral, or 
socially disapproved behaviors. Social justice scholars should especially pay attention to social 
desirability issues when they study sensitive issues relating to power, injustices, and abuse. 

First, researchers should do all they can to ensure confidentiality, if not anonymity, in 
research, which will help alleviate social desirability bias. Allowing respondents to fill out 
surveys privately may help a great deal, as well. Also, researchers should do their best to use 
a non-judgmental approach to item construction in survey research so that participants feel 
at ease while sharing their lived experiences. For instance, instead of asking, “Have you ever 
stolen anything from a store?,’ it might be better to rephrase the item to make the response 
more socially agreeable by asking, “Have you ever had to take something from a store without 
paying for it?” (Dillman, 2005, p. 75). 

Another related issue with social desirability is dealing with fence-sitters. These are 
respondents with a tendency to pick a neutral midpoint nearly anytime they are asked to 
respond to an item. They do not feel strongly about any issue in a negative or positive way. 
Having an explicit middle point in the scale or an option such as “no opinion” for such 
respondents is a good idea as it increases reliability. However, the more problematic type of 
respondents are the floaters. They do the opposite of fence-sitters. Although they might 
not know anything about an issue or have no opinion, they still go ahead and pick an answer 
option. To avoid this type of error, it is advisable to include a “don’t know” answer option in 
items. On the other hand, some researchers prefer forced-choice items, which specifically 
avoid the “don’t know” option. They do this if they get a sense that people would pick it as 
an easy way out without committing to an opinion (Schutt, 2015). 


Deciding the Order and Presentation of Selected Items 


So far, we have focused on individual items and answer options in isolation. However, survey 
items and response choices are all part of a larger questionnaire instrument. A questionnaire 
is an instrument or form that contains items and answer options and is administered to a group 
of people to collect information for surveys. Items and answer choices should be seen in the 
context of how the questionnaire itself is designed as a whole and where each item fits within 
it. Items that are irrelevant should be dropped. 


Organizing the Items in the Right Order 


Researchers should pay attention to the order of the items, which influences how the items 
are understood and answered. The survey should be divided into broad thematic sections. 
Each section of the questionnaire should have a clear focus. All items pertinent to that section 
should be asked in the same place before moving on to another topic. The sections should be 
ordered in a logical manner, preferably as close as possible to how a natural conversation on 
such a topic might progress. Each section generally has its own introduction and guidelines 
to the respondent for how to go about providing responses. For instance, a section might be 
introduced to the respondent by saying, “The next section of this survey asks you about your 
views of your employer’s commitment to diversity, equity, and inclusion. For each of the fol- 
lowing items, please indicate your agreement with the statement on a scale of 1 = strongly 
disagree to 5 = strongly agree.” 
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In general it is best not to start a survey with boring, challenging, or sensitive questions. 
For instance, although demographic questions such as asking people to report their gender, 
race, income, and age might be easy to answer, many respondents consider them rather per- 
sonal. Therefore, in general, demographic questions are better off placed at the end of the 
survey rather than in the beginning. Best practice is to begin a survey with a set of items that 
respondents will find interesting and engaging. 

It is important not to waste respondents’ time and energy by asking them several items that 
are not relevant to them. One way to do this is to include screening or filter questions. 
Filter questions are items that can help identify which subgroup of the sample should be asked 
a set of questions. For instance, if there is a set of items on marital violence, a filter item could 
be one that asks the respondents whether they are married. For those who say “Yes,” they will 
go ahead and answer the items in that subsection. Others who are not married skip to the 
next unit. These items that are only asked of one subset of the respondents are called contin- 
gency questions. By combining filter questions and contingency questions, the researcher 
creates a unique pattern of order in which each subgroup answers the survey. This is called 
a skip pattern. Be sure to clearly indicate to respondents which items to skip. This can be 
done automatically in web-based and electronic surveys. In paper-and-pencil surveys, it helps 
to use arrows and other aids to help the respondent navigate the questionnaire. Having too 
many back-and-forth skip patterns can annoy and confuse respondents. They should be used 
thoughtfully and sparingly. 

To increase the internal consistency of a concept measured in a study, it is best that the con- 
cept is measured using more than one item, especially if concepts have multiple dimensions 
and are umbrella terms that include other related terms, as discussed in Chapter 3. A scale 
or index is a set of items that hang together to measure a concept. This creates a compos- 
ite measure where the average response across the set of items is developed. Using multiple 
items to measure a concept allows researchers to include different items that each “get at” the 
concept from a slightly different perspective. Researchers can use established scales (those avail- 
able in the research literature), modify existing scales, or create scales on their own. Whenever 
possible, it is better to use existing indices rather than to start creating one from scratch. Pre- 
existing scales have likely already been tested across various contexts for reliability and validity. 
However, it is also possible that the existing scales are not directly relevant or appropriate for 
the population that you are studying. They can either be modified to suit your specific project, 
or you might want to try your hand at creating your own items that will do a better job. 

The Ramasubramanian (2010) study that measured how perceptions of televised racial 
stereotypes influenced policy opinions is an example of a study that used existing items to 
measure a central concept in the study. One of the key variables was support for affirmative 
action, which was measured using a scale: 


Support for affirmative action policies. Three items used by past research to measure opinions 
regarding government policies on affirmative action were included in the present study. 
A seven-point Likert-type scale, ranging from strongly disagree to strongly agree, was 
used. Higher scores indicated greater support for pro-minority policies such as affirma- 
tive action. Three statements measured attitudes toward affirmative action: “Affirma- 
tive action is one good way to deal with past racial discrimination;” “Affirmative action 
policies are unfair to White participants” (reverse-coded); and “Universities should not 
consider an applicant’s race in the admission process” (reverse-coded). 


(p. 111) 


You will notice that two of the three items in the study described were reverse-coded. Let us 
explain what that means. When combining items to form an index, it is important to consider 
response style bias. This is the tendency of the respondent to go down a list of items indicating 
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Figure 5.5 Make the questionnaire simple and accessible to complete 


Source: Paul Townsend, www.flickr.com/photos/brizzlebornandbred/41524990695 


the same response (say option one on a seven-point scale) without necessarily taking the time to 
read or understand each item. To avoid this, researchers should keep items short and to the point. 
They could also reverse the polarity of some of the items in the scale (also known as reverse 
coding). What this means is that instead of all items in an index going from positive to nega- 
tive, some scales could be reversed to go from negative to positive. Doing so helps the researcher 
to check whether the answers have been consistent for a given respondent. For example, if one 
item asks how they feel in the workplace ranging from “1—not at all safe” to “5—extremely safe,” 
another item with reversed polarity of the scale could range from “‘l—extremely comfortable” to 
“5—not at all comfortable.” If they are consistent in their responses, the respondents who score 
higher numbers on the safety question are expected to score lower numbers on the comfort ques- 
tion since safety and comfort often go hand in hand in the workplace. 


Making the Questionnaire Accessible 


In making surveys accessible, the principles of universal design become relevant. Universal 
design focuses on equity, accessibility, and diversity, all of which are important ethical principles 
for social justice researchers (Connell et al., 1997). By designing for the groups that have the 
most difficulty in accessing products or spaces, the design is made more usable for everybody. 
An accessible and inclusive questionnaire will reduce errors and biases across all partici- 
pants. A visually attractive and easy-to-follow questionnaire is likely to improve response 
rates and the quality of answers. Labeling each section clearly, numbering the items and 
pages, providing clear instructions on how to answer each item, and using an open layout 
are crucial, especially in self-administered questionnaires. For example, the use of fairly large 
font size and plenty of white space in between items helps not just those with poor eyesight 
but all respondents. The visual appearance and clarity can be further enhanced by using font 
sizes and bold formatting where appropriate. Some surveys use bolder fonts for the items and 
lighter ones for the answer options. When there are multiple items in the scale, presenting 
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them in alternating dark and light shades or colors helps respondents ensure that they are 
answering the correct item. Skip patterns should be used infrequently and should be indicated 
clearly through arrows and instructions. 

Using simple language that is understandable for those with low levels of education, lit- 
eracy, or limited fluency makes the survey simpler for everyone completing it. Avoiding the 
use of jargon and technical terms provides greater clarity and appeals to the simplicity and 
equitable aspects of design. An example of flexibility in design is the use of online surveys 
that are self-paced rather than timed, allowing respondents to adapt the survey to their own 
pace. Making the survey available in a computerized format could also help make it available 
in various modes such as visual, textual, and graphic such that it can be understood through 
a variety of sensory modes. 


Research in Action 


The Health Information National Trends Survey (HINTS) is a national survey of adults 
aged 18 or older in the U.S. Administered in English and Spanish, this survey has been 
conducted since 2003 by the National Cancer Institute. Their website provides survey 
questions, data, and other supporting documents to the public (http://hints.cancer. 
gov). Figure 5.6 shows a sample page from the HINTS Cycle 2 English version from 
2018. The annotations give information about the variable name assigned to the item 
and the numerical values to the answer options for data collection. The use of white 
space and visual formatting of the items makes them easy to follow and makes it easy to 
respond. Key points in the items are underlined for added emphasis. 


C: Your Health Care C4. The following questions are about your 
communication with all doctors, nurses, or 
. : ss other health professionals you saw during 
C1. Not including psychiatrists and other the past 12 months. 
mental health professionals, is there a g 
particular doctor, nurse, or other health How often did they do g E 
professional that you see most often? each of the following? = 3 S è 
tegularProvider 
Yes ; wwwwes 
No a. eee chance to ask all 
_ e health-related questions you 
p. qunncekskaiest fous o on CHEENA 
C2. Inthe past 12 months, not counting times i e Laon a E) 
you went to an emergency room, how eeir enned 
many times did you go to a doctor, nurse, sat Mee cate cai rence 
or other health professional to get care for You Wanted ecu E tare B| 
you rself? InvolvedDecisions 
FreqcoProvider d; Maka eae you ante ae 
= ings you n o o take 
None æ GO TO C6 on the next page care of your health...................:.0+ 
E| 1 time Explain ;codlextatene 
j e. ain things in a way you 
E] 2times could understand. B| ml 
3 times ExplainedClearl 
; f. Spend enough time with you........ a o 
E] 4 times SpentEnoughTine | 
n $ g. Help you deal feelings 
5-9 times uncertainty about your health or 
10 or more times B m 


Figure 5.6 HINTS survey 


Source: Health Information National Trends Survey ® U.S. Department of Health and Human Services. https:// 
hints.cancer.gov/data/survey-instruments.aspx 
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Pretesting Questionnaires 


Feedback is an integral part of survey design; the more the better. While the principles of 
designing appropriate questions discussed earlier will greatly improve the quality of the ques- 
tionnaire, getting external feedback before you actually launch your survey research study is 
invaluable. You might think that you have given clear instructions and that all the items are 
perfectly understandable. However, working with actual respondents in real-world settings 
can quickly change this notion. 

Before seeking external feedback, researchers should fill out their questionnaire them- 
selves, note any challenges and errors, and make corrections. Sometimes researchers will 
assemble a panel of experts to provide feedback on the reliability, validity, and cultural appro- 
priateness of the measures being used. Getting feedback from key stakeholders such as com- 
munity leaders, industry representatives, government agencies, and elected representatives 
is especially important when conducting community-based survey projects. However, it is 
not always feasible to organize a panel of experts and community members. In that case, 
researchers could have friends, colleagues, or family members complete the questionnaires 
for initial feedback. 

Discussing and receiving feedback on the way you have measured key variables on your 
questionnaire ahead of time helps ensure that the items that are included are relevant and 
appropriate to the shared objectives and goals of the community. Another immensely use- 
ful approach is to compare your hypotheses and research questions against the items you 
have included in your questionnaire. That way, you will make sure that you have not missed 
anything and that you will have the data you need to test your hypotheses and explore your 
research questions. 

The best way to find out whether a questionnaire works is to give it to a small group of 
respondents and find out. A typical pretest is conducted with a small group of participants 
(maybe 10 to 15 people) as similar to the population as possible. After completing the pretest, 
respondents are given some space to write in comments and/or time to discuss their experi- 
ence completing the survey with the researchers. If the items are not being understood as 
intended or they appear to be challenging for whatever reason, this is the perfect time to 
consider what revisions and modifications would bring greater clarity and consistency. It is 
best to administer the survey using the same mode as what is intended eventually—via phone, 
in person, online, and so on. Even if there is no time to complete the entire questionnaire in 
the pretest, researchers could consider testing some sections, especially those that are newly 
developed or particularly difficult to understand. 

There are also somewhat more ambitious and time-consuming approaches to pretesting 
questionnaires. One form of feedback is obtained by using focus group discussions with a 
subset of the population that is being examined. By working directly with potential respond- 
ents, such discussions can help get important information about whether the key variables 
are easily and consistently understood by everyone. By listening and being attentive to both 
verbal and nonverbal cues in such discussions, researchers can also get crucial insights about 
whether the items are too complicated, biased, sensitive, confusing, irrelevant, effortful, or 
time-consuming. A related method is called cognitive interviewing in which participants 
are asked to describe their thoughts and feelings aloud as they work on completing the ques- 
tionnaire. The respondent answers the actual survey item and then is asked a follow-up item 
to probe about how the item was understood and to identify any challenges that the item 
posed for the respondent. 
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Deciding How to Administer Surveys 


There are multiple modes in which surveys can be administered. These can be broadly catego- 
rized as interviewer-administered and self-administered surveys. Interviewer-administered 
surveys are those in which the survey is administered, at least in part, by a survey interviewer. 
These can either be face-to-face interviews or telephone interviews. Self-administered 
surveys are those that are administered by the respondents themselves without any aid from 
a survey interviewer. These can be done through mail surveys or online surveys. In general, 
self-administered surveys are much cheaper compared to interviewer-administered ones. 


Personal or Face-to-Face Interviews 


Personal or face-to-face interviews are interviewer-administered surveys done in person. 
They have the highest response rate. These days, most interviewers use laptops or other 
mobile devices instead of paper-and-pencil questionnaires to input the data they collect. Face- 
to-face interviewing is especially suitable when the population is not literate or does not have 
regular access to phones and computers. Interviewers are able to ask open-ended questions, 
answer or clarify any questions that respondents might have, and engage the respondents for 
longer periods of time compared to other data-gathering modes. 

However, personal interviews are time-consuming and expensive. There could also be 
interviewer effects where the respondents are influenced in how they answer the items by 
who the interviewer is, such as their gender expression, ethnicity, or social class. Another 
related problem arises if the interviewer is not consistent or systematic in how the questions 
are being asked of respondents. Interviewers need to be trained adequately, especially if they 
are not familiar with the local customs and cultural norms of the community being studied. 
Recruiting qualified interviewers can be a challenge. 

Another issue to consider is interviewer safety. When working with social justice issues, 
this could mean anything from physically challenging terrains to political unrest to high crime 
areas. For some research projects, interviewers work in pairs or travel in groups with a security 
guard or local trusted person from the community. Finding accommodation, transportation, 
and reliable communication methods are crucial. Interviewers might need to first establish 
themselves as legitimate researchers by gaining the approval of authority figures or community 
leaders before starting the survey. It is also important to ensure that the interviews are con- 
ducted in a private setting to protect the confidentiality of the respondents, especially when 
discussing potentially sensitive topics. 


Telephone Interviews 


Telephone interviews are also interviewer-administered, but instead of face-to-face inter- 
actions, the mode of researcher-respondent interaction uses a landline or mobile phone. 
These days most telephone surveys are completed using computers. This method is referred 
to as computer-assisted telephone interviewing (CATI), which allows for such aspects as 
random selection of phone numbers, automatic dialing, and inputting responses directly 
into software. Similar to personal interviewing, telephone interviews are also excellent for 
open-ended questions that need further probing. It is much more cost-effective and faster 
than face-to-face interviews. Interviewers do not have to worry about accommodation, 
meals, or safety. 
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Yet, a phone survey is only meaningful if phone access is relatively high in the target 
group, which could be an issue among some under-resourced populations. Depending on 
whether a full sampling frame is available or not, it might be hard to determine how proba- 
bilistic the sample is. As we learned in Chapter 4, in order to draw a probability sample, the 
researcher needs access to a comprehensive list of all potential survey respondents. It is dif- 
ficult to know whether a list of all phone users would be available in any particular location, 
and any potential list might well be limited by not including those with private numbers, 
for instance. The response rates are lower and the length of interviews are shorter compared 
to personal interviews. With the rise of spam calls and telemarketers, those doing legitimate 
surveys find it hard to reach respondents, who either fail to pick up the phone or hang up 
on the interviewer. 


Mail Surveys 


Mail surveys are a type of self-administered method where the questionnaire is mailed to the 
respondents, who complete it and mail it back to the researchers. It is an appropriate method 
when sensitive items are being asked that are best answered privately, which is not possible 
with interviewer-administered surveys. They are cost effective although time consuming. 
Without the assistance of an interviewer, there is no one to explain or clarify instructions, 
which is especially problematic among populations with low literacy levels. Mail surveys can 
also easily get misplaced, forgotten, or not mailed back even when completed. It is also hard 
to identify who within a household actually completed the survey. 


Online Surveys 


Online surveys (also known as web-based or electronic surveys) are the most recent forms of 
self-administered surveys. Here respondents are directed to a website through a link, where 
they complete the survey on a computer, tablet, smartphone, or other electronic device. 
Surveys are also being conducted through social media and text messages. Tools such as Sur- 
vey Monkey, Google Forms, and Qualtrics are used by several researchers to design online 
surveys that can be embedded into websites, social media posts, emails, and text messages. 
Another recent and important development in Communication survey research is the use 
of Amazon’s Mechanical Turk (or MTurk for short). This online space pairs “requesters” 
and “workers” on short-term tasks, including completing brief surveys for a small monetary 
incentive. 

Online surveys can accommodate a variety of interactive and multimedia elements such as 
visuals, embedded clips from media content, graphics, pull-down menus, and slider bars. For 
example, the respondent can be prompted if they missed an item inadvertently or can even 
pause and resume the survey at a later point in time. Once the survey has been created online, 
it can be sent simultaneously to any number of respondents. It is, therefore, an inexpensive 
and efficient method of data collection. 

On the other hand, if there is only limited access to electronic devices, limited bandwidth 
or intermittent connection to the internet among the population, this is not an appropriate 
method. Online surveys should also be compliant across multiple platforms, browsers, and 
devices so that all respondents can have access to them. Without the ability to check exactly 
who is participating in online surveys, it is not possible to know whether the respondents 
were unique. Also, without an interviewer administering the items, sometimes online surveys 
yield a lot of missing data and response-style biases, suggesting that the respondents did not 
bother to read the items and answer options carefully while answering. 
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Finding Ways to Improve Survey Response Rates 


One of the biggest concerns for survey researchers is the response rate, which is the per- 
centage of completed surveys as a function of the number of surveys that were distributed. 
For instance, if you distributed 100 surveys and received 80 back from the respondents, 
your response rate is 80%. As we saw in the chapter on sampling, when the response rate 
is low, the main concern is that the sample is not generalizable to the population from 
which it is being drawn. It is important to analyze the size and the composition of non- 
responses to understand whether there are systemic patterns so they can be corrected. For 
instance, if we notice a systematic pattern that most of the nonresponses are from a certain 
geographic area where the majority are Spanish rather than English speakers, a corrective 
measure could be to translate the survey into Spanish as well. There are several reasons 
why respondents might not engage with a survey. These can include poor sample cover- 
age, noncontact, nonresponse, or inability to participate. Each is explained in detail in the 
text that follows. 

Poor sample coverage could lead to sampling errors if the original sampling frame was 
inadequate. This can happen if the respondents have moved to another address or changed 
their phone number, which is not reflected in the sampling frame. Email addresses could also 
become defunct, especially if they are associated with a particular institution or organization, 
which the respondent has now left. So, alas, right at the outset, the sampling frame is leav- 
ing out a portion of the population in a systematic way, which becomes difficult to correct 
because there is no way that the respondents will even receive the invitation to participate in 
the survey. It is important to make sure that the sampling frame is as up-to-date and accurate 
as possible before beginning the study. 

Another issue is noncontact, which means that some individuals simply may not be 
able to be reached, although they were a part of the sampling frame. When the population 
spans multiple regions, nations, and cultures, survey design becomes a complex endeavor. 
Factors such as political upheavals, religious holidays, weather conditions, and varying 
customs and norms can influence survey dissemination. Populations of interest could also 
be living in fairly difficult terrains that are hard to reach in certain seasons and severe 
weather conditions. They could also be migratory populations who move across geo- 
graphic regions due to weather or seasonal occupations. These circumstances could pro- 
vide unique challenges to be overcome by survey researchers. If the social justice project 
happens to involve groups such as farmworkers, miners, construction workers, or domestic 
workers, researchers should consider that they might be away from family for their work 
for long periods of time. 

Even if contact is made, there can be nonresponse, which means that the individuals 
decline the request to participate in the survey (unit nonresponse) or refuse to answer spe- 
cific items on the survey (item nonresponse). Unit nonresponse refers to those who were in 
the sampling frame, were contacted, but either refused to answer the entire survey or portions 
of it. Sometimes, the specific time or location might not be convenient for the respondents 
to complete the survey. We all can think of a time when we were asked to complete a phone 
survey or paper survey at an inconvenient time and had to refuse participation. It could also 
be that some respondents simply decline participation all the time, regardless of what the sur- 
vey is about and how many times they are contacted. This could be a result of prior negative 
experiences with surveys where they felt that they wasted their time or their input was not 
taken seriously. Even if multiple contacts are made, some respondents simply ignore or decline 
participation in the survey. They might also simply be unable to participate, say, due to mental 
or physical health conditions. 
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It is also important to remember that groups that might be of interest to social justice sur- 
vey researchers might be overworked, with multiple jobs, multiple shifts, and perhaps long 
commutes. Within this context, completing surveys is a low priority compared to meeting 
more basic needs like food, safety, and shelter. With phone surveys, for instance, this could 
mean the researcher must call back several times, perhaps during different times of the day 
or week, in order to overcome initial nonresponse. Interviewers must understand that their 
phone calls might be interrupting work or that the participant might not be able to give their 
full attention to their cell phones since they may be in a crowded area or driving somewhere. 
The increase in spam calls and junk messages also means that potential respondents might not 
answer calls from numbers they do not recognize. With mailed surveys, one could send an ini- 
tial letter of introduction, reminder postcard, replacement surveys in case some are misplaced, 
and a thank you card for those who completed. If research funding allows, the researcher 
could also pay for postage. 

In interviewer-administered surveys, if the interviewer is perceived to be unfriendly, 
unprofessional, rude, or uninterested, the respondents might refuse participation. The proto- 
cols that guide interviewer-respondent interactions, be they administered in person or phone 
or any other mode, should be polite, professional, and personalized. Using specific names to 
address participants in the cover letter or email that includes the online survey link is always 
more likely to garner response as opposed to a generic invitation. Assurances of confidenti- 
ality and privacy are another way in which researchers can communicate that they take this 
responsibility seriously. If possible, researchers should offer monetary or other incentives such 
as extra credit to college students or a random, lottery-type drawing for an item of interest 
to the participants. Researchers also may use multiple modes of data collection to reach the 
respondent. For example, they might send a mailed survey but follow up via phone to remind 
prospective research participants about it or send an email with an online web version of the 
mailed survey. 

Another form of nonresponse is item nonresponse, where respondents do not answer certain 
items on the questionnaire. In self-administered surveys, it could simply be that respond- 
ents failed to notice a particular item. At other times, the respondents might notice and 
read the items but have difficulty in understanding them. Avoiding confusing and hard-to- 
understand items becomes crucial here. Items that are seen as sensitive, too personal, embar- 
rassing, or intimidating are also likely to lead to nonresponse. For example, topics such as 
domestic violence or sexual harassment might not only be emotionally difficult to answer 
but could sometimes be physically dangerous to respond to, for fear of retaliation, if privacy 
is breached. Avoiding answering such items could be a psychological coping mechanism for 
some respondents. Many of the issues that social justice researchers are interested in are likely 
to fall into this realm, which is why particular attention has to be paid to ensure the safety, 
comfort, confidentiality, and privacy of the participants. 

It is also common for respondents to only complete items that require less effort, such as 
close-ended items as responding to open-ended items might take more time and effort. As 
we have discussed earlier, some respondents might simply not know the answers or be able 
to recall them. Finally, it is possible that some participants do not take the survey seriously 
because they do not see the value of completing it but might be interested in compensation 
such as extra credit. Under these circumstances, they might either skip through items or 
answer them without reading them carefully. If the survey participant does not see the survey 
as meaningful, relevant, and helpful, they might reluctantly agree to participate but leave a lot 
of data missing. 


Soliciting Opinions Through Research 121 


Scholar Spotlight: Dr. Sasha Costanza-Chock 


Figure 5.7 Scholar Spotlight: Dr. Sasha Costanza-Chock 


Sasha Costanza-Chock (pronouns: they/them or she/her) is a scholar, activist, and 
media-maker, and currently Associate Professor of Civic Media at MIT. They are a 
faculty associate at the Berkman-Klein Center for Internet & Society at Harvard Uni- 
versity, faculty affiliate with the MIT Open Documentary Lab and the MIT Center for 
Civic Media, and creator of the MIT Codesign Studio (codesign.mit.edu). Their work 
focuses on social movements, transformative media organizing, and design justice. Dr. 
Costanza-Chock’s first book, Out of the Shadows, Into the Streets: Transmedia Organizing 
and the Immigrant Rights Movement was published by the MIT Press in 2014. Their new 
book, Design Justice: Community-Led Practices to Build the Worlds We Need was published 
by the MIT Press in 2020. They are a steering committee member of the Design Jus- 
tice Network (designjusticenetwork.org) and a board member of Allied Media Pro- 
jects (AMP); AMP convenes the annual Allied Media Conference and cultivates media 
strategies for a more just, creative, and collaborative world (alliedmedia.org). While 
Dr. Costanza-Chock uses multiple research methods in their work, in this chapter, we 
showcase their use of survey research within activist-scholarship. 


Q: You use a variety of research methods in your work. What do you think is the role of quan- 
titative methods in social justice scholarship in communication? 

Dr. Costanza-Chock: I locate my work within the long history of participatory action 
research (PAR). PAR is a framework with roots in the work of scholars such 
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as Kurt Lewin, John Dewey, and (later) Paolo Freire, Orlando Fals-Borda, and 
Linda Tuhiwai Smith that emphasizes the development of communities of shared 
inquiry and action. In PAR, we consider communities to be co-researchers, rather 
than solely “research subjects.” It’s a research stance from which we can employ 
any methods, including quantitative methods. So, for me, social justice scholarship 
in communication, regardless of the methods employed, is about partnering with 
community-based organizations from the beginning of the project. Together we 
can define research questions, choose and execute methods, analyze what we find, 
and make recommendations based on the research. 

Q: What are some topics that you have studied using survey methods? Can you tell us a little 
about those studies? 

Dr. Costanza-Chock: Some of the projects I’ve worked on that involved surveys 
include the Occupy Research Network and the Transformative Media Organ- 
izing Project. The Occupy Research Network was an open, shared space for 
researchers who participated in the Occupy Movement. We produced the 
Occupy Research General Demographic and Political Participation Survey 
(ORGS), with 5,074 individual respondents; it was quite influential among 
scholars interested in Occupy. Over 100 peer-reviewed articles and books cite 
the ORGS dataset, while nearly 200 cite “Occupy Research.” Some of the 
key findings from that survey are summarized in a chapter in the book We Are 
Many (2012). I also drew on the survey data in my article (Costanza-Chock, 
2012). The Transformative Media Organizing Project was a multiyear study of 
LGBTQ and Two-Spirit media advocacy practices, funded by the Ford Foun- 
dation. This project included a series of expert interviews and an organiza- 
tional survey with 231 respondents. We developed research questions together 
with a group of partner organizations from all around the U.S.A.; together we 
produced an overview of how the organizations that work to improve the lives 
of LGBTQ and Two-Spirit people currently utilize media and made recom- 
mendations to funders as to how they might better support media activism in 
this space. Our findings are available at transformativemedia.cc, and in Media, 
Culture & Society (Costanza-Chock, S., C. Schweidler, & the Transformative 
Media Organizing Project, 2017). 

Q: One of your expertise areas is design justice. Can you tell us more about this concept? How 
can it be applied to the design of surveys and questionnaires? 

Dr. Costanza-Chock: To quote from a recent paper I presented at Design Research 
Society: “Design is key to our collective liberation, but most design processes 
today reproduce inequalities structured by what Black feminist scholars call the 
matrix of domination. Intersecting inequalities are manifest at all levels of the 
design process. [Design justice] builds upon the Design Justice Principles, devel- 
oped by an emerging network of designers and community organizers, to propose 
a working definition of design justice: Design justice is a field of theory and prac- 
tice that is concerned with how the design of objects and systems influences the 
distribution of risks, harms, and benefits among various groups of people. Design 
justice focuses on the ways that design reproduces, is reproduced by, and/or chal- 
lenges the matrix of domination (white supremacy, heteropatriarchy, capitalism, 
and settler colonialism). 
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Design justice is also a growing social movement that aims to ensure a more 
equitable distribution of design’s benefits and burdens; fair and meaningful par- 
ticipation in design decisions; and recognition of community-based design tradi- 
tions, knowledge, and practices” (Costanza-Chock, 2018a). Applied to survey 
design, design justice means that instead of researchers creating surveys to study 
communities only as research subjects, leaders from community-based organiza- 
tions (CBOs) that already exist to advance the interests of marginalized commu- 
nities should be included as research partners from the beginning. That means 
that community-based organizations participate with the researchers to create the 
research questions, select methods, craft survey design, do data collection, analyze 
the findings, draw conclusions, and publish the results. 

Q: A common challenge in doing survey research is nonresponse. What are some ways you 
suggest for improving response rates, especially while working with marginalized groups? 
Dr. Costanza-Chock: In my experience, partnering with community-based organiza- 
tions that are known to have strong accountability mechanisms to marginalized 
groups is key to survey response. Communities are tired of having their infor- 
mation mined and extracted. Why should marginalized communities give their 
information to yet another set of researchers? In research partnerships with trusted 
CBOs, people are more likely to have a sense that their information will be used 

to advance their interests. 

Q: You identify both as a scholar and activist. What does activism mean to you as a commu- 
nication researcher? Do you have any advice for budding scholar-activists? 

Dr. Costanza-Chock: All research is political; the question is, are you willing to 
explicitly talk about the politics that guide your work? To me, partnering with 
community-based organizations and social movement organizations has always 
been the most interesting, productive, grounded, and ethical way to work. For 
budding scholar-activists, I recommend you find community-based organizations 
whose values and ways of working you really resonate with, develop a relation- 
ship, and figure out how your research skills can be useful to the community. It 
will be hard, but very rewarding, and it will ground your scholarship in real-world 
problems, guide you toward useful insights, and ultimately, you’ll learn so much! 


Summary 


Surveys are versatile, easy, cost-effective, and efficient methods for social scientists. They can 
be used to gain knowledge about the attitudes, opinions, beliefs, knowledge, perceptions, 
intentions, and behaviors of individuals. They help observe key patterns of relationships 
among variables to understand complex social issues. They allow researchers to examine 
cross-population generalizability and cross-cultural patterns in data. By collecting data on the 
views, thoughts, experiences, and concerns of groups of respondents, researchers can amplify 
diverse voices. 

Cross-sectional surveys are one-time surveys, whereas longitudinal surveys are administered 
more than once over a period of time. Longitudinal surveys could be trend, cohort, or panel 
designs. Longitudinal designs help to capture data across time, which could be valuable for 
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social justice researchers to be able to track changes in social issues over time. However, these 
surveys are expensive and there could be attrition. Apart from designing their own surveys, 
researchers could also use research firms, omnibus surveys, or publicly available datasets for 
their projects. 

Some ways to increase the validity and reliability of surveys are by reducing measurement 
error, improving sample coverage, and increasing response rates. Researchers should ask clear 
and relevant items that are consistently understood by all participants. They should avoid ask- 
ing confusing, biased, double-negative, and double-barreled items. They should not include 
items for which respondents do not possess knowledge or cannot access answers easily. The 
answer options should be meaningful, collectively exhaustive, and mutually exclusive so that 
the respondents can accurately record their responses. Social desirability should be minimized 
by emphasizing privacy, confidentiality, and non-judgment from researchers. 

Items should be organized in a logical sequence in questionnaires. Using filter questions 
and skip patterns help participants to only respond to questions that are relevant to them. 
Where appropriate, the survey should be made available across different languages, modes 
of administration, and communication devices/platforms. Interviewer-administered surveys 
can be conducted face-to-face or over the phone. Self-administered surveys can be done via 
mailed surveys or electronically. A growing concern in survey research is nonresponse. In gen- 
eral, surveys that assure confidentiality, ask relevant questions, are personalized, provide incen- 
tives, and are perceived as leading to actionable outcomes are more likely to be completed. 

When designed with care, surveys are an excellent tool for quantitative researchers inter- 
ested in advancing social justice. Well-designed surveys for social justice seek to find relevant 
information by allowing participants to share confidentially and assure them that the findings 
will be used for actionable social change. 


Discussion Questions and Activities 


1. Discuss the key benefits and challenges of using survey methods, especially for projects 
oriented toward social justice initiatives. 

2. What are the most important ethical considerations in survey research when working on 
social justice issues? 

3. Using a topic of interest to you, explain how you would conduct a cross-sectional survey, 
trend, cohort, and panel study. Who might serve as a population of interest for each type? 
How do the types differ from one another? 

4. Sometimes we learn best by making mistakes. Can you give examples of poorly writ- 
ten survey items (those that are biased, double-barreled, include double negatives, and 
include answer options that are not mutually exclusive and collectively exhaustive)? Then, 
for each, correct these mistakes by providing a better survey item. 

5. How can surveys be designed to be more attractive and accessible to respondents? 

6. Discuss the advantages and disadvantages of the various modes of data collection—mail, 
phone, in person, or online—one can use in survey research. 
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6 Studying Responses Through 
Experimental Methods 


Experiments have always been central to scientific research. They are often considered the 
defining component of the social scientific method of inquiry (Webster & Sell, 2014). Some 
famous scientific experiments are the water displacement experiment by Archimedes after 
which he ran into the streets yelling “Eureka! Eureka!,” Galileo Galilei dropping a heavy and 
a light ball from the Leaning Tower of Pisa to show that they land at the same time, and Ivan 
Pavlov’s classical conditioning experiments where dogs start salivating every time a bell was 
rung or music was played because of associations with food. More recent social justice-ori- 
ented experiments include Jane Elliot’s classroom experiment where she labeled the blue-eyed 
children as superior and gave them extra privileges on the first day and then reversed roles 
the next day with brown-eyed kids to teach about the negative effects of racial discrimina- 
tion (Figure 6.1). Another famous experiment is the bobo doll experiment by Albert Ban- 
dura, where he showed that kids were more likely to learn aggression through imitation than 
because of genetically inherited traits. This experiment is especially relevant to Communica- 
tion scholars who study social learning and violence. 

Several of these experiments led to breakthroughs in scientific knowledge and theory 
building while also dealing with real-world social issues. As we saw in Chapter 1, basic 
research focuses on theorizing knowledge, whereas applied research focuses on solving prac- 
tical problems. Experiments can contribute to both basic research through theory building 
and to applied research through extending theories into practical solutions. As you will recall, 
a research hypothesis is a specific testable proposition of a relationship between variables 
put forth by the researcher. Experiments are excellent methods to test research hypoth- 
eses because they allow researchers to observe differences across groups to see if a proposed 
hypothesis holds or not. You will learn more about hypothesis testing using statistical methods 
in Chapters 8 and 9. 

Whereas surveys provide an overall picture of any social issue of interest to the researcher, 
experimental methods are helpful in studying cause-effect relationships among variables. In 
fact, experiments are seen as a gold standard method for establishing cause and effect relation- 
ships between variables. Therefore, when the research design calls for examining whether 
one variable is affected by another, experiments are the best choice (Webster & Sell, 2014). 
The meaning of causation and the specific requirements for establishing causal relationships 
among variables will be discussed later in this chapter. Not only do experiments help identify 
causal effects, but they are also a powerful means for researchers to understand the theo- 
retical mechanisms underlying a causal relationship. That is, they explain the process and 
the mechanism about why variables are related as the cause and effect. Thus, they help to test 
scientific theories and theoretical processes using empirical data-driven methods. 

For social justice research, in particular, the experiment is likely to be the go-to method 
for studying the effects of an intervention. Does watching a media literacy video increase 
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Figure 6.1 Jane Elliot’s blue eyed/brown eyed classroom experiment 


Source: Film: “A Class Divided” 


critical thinking about stereotypical media depictions among teens? Does seeing a documen- 
tary on food deserts in under-resourced communities spark monetary giving to a campaign to 
bring in a local grocery store? Questions such as these that pertain to whether exposure to or 
participation in a program affects individuals are well suited for experimental methodology. 
Intervening in social injustices and inequalities is at the heart of social justice research, and 
experiments are particularly useful to test the outcomes associated with specific interventions. 


Independent, Dependent, Control, and Confounding Variables 


In experiments, the independent variables (or cause or predictor variable) are manipu- 
lated or introduced into the setting by the experimenter. Manipulating a variable means the 
researcher designs the variable so that different participants in the study experience that same 
variable differently. The dependent variable (or effect or outcome variable) is the out- 
come variable that is measured. It is the variable that the researcher thinks will be affected by 
the independent variable. 

For example, imagine a study in which a researcher examines how exposure to posi- 
tive portrayals of Black women in video games leads to lower levels of racialized sexism. 
The independent variable is exposure to positive portrayals of Black women in games. The 
dependent variable is the level of racialized sexism. The researcher would manipulate the 
independent variable to create two groups: one that plays a game with positive portrayals of 
Black women and another that plays a game without such content. The first group would be 
called the experimental or treatment group since it receives the treatment. The second 
group would be called the control group, which does not receive any treatment. The two 
groups, then, experience the variable—exposure to positive portrayals of Black women in 
games—differently. 
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Figure 6.2 Independent variable, dependent variable, and confounding variable 


Source: Emily Riewestahl 


Apart from the independent and dependent variables, there are other variables that need 
to be considered in an experiment. All these other variables other than the independent and 
dependent variables are called extraneous (or third) variables. There are two types of 
extraneous variables: control variables and confounding variables. When extraneous variables 
have been accounted for by the researcher, they are called control variables. Extraneous 
variables that have not been controlled for are called confounding variables. Researchers 
need to minimize confounding variables by controlling for as many relevant extraneous vari- 
ables as they can. Researchers try to control the design of the experiment to reduce the num- 
ber of possible confounding variables. Researchers also measure control variables, as we will 
explain later in the chapter, to test whether the relationship between the independent variable 
and the dependent variable holds even when taking these additional variables into account. 

In the gaming study discussed earlier, for example, some control variables could be the 
amount of time the game is played, the device used, the room where the game is played, 
the time of day when the game is played, and so on. When the researcher fails to control for 
an important extraneous variable, that is a confounding variable. For instance, imagine that 
the game the researcher chooses for participants to play in the experimental group not only 
contains more positive portrayals of Black women, but it is also more violent than the game 
the researcher chooses for participants to play in the control group. The level of violence in 
the game is a confounding variable. We do not know whether the changes in the dependent 
variable (level of racialized sexism) are caused by the independent variable (exposure to posi- 
tive portrayals of Black women) or the confounding variable (level of violence) (Figure 6.2). 


Role of Experiments in Establishing Causation and Causal 
Mechanisms 


As we saw earlier, one of the biggest advantages of experiments is that researchers can infer 
causation from them. But what does “inferring causation” mean? Here are the criteria for 
establishing causality (Schutt, 2015): 


1. Association: First, there should be a significant association between the independent 
variable and the dependent variable. In Chapter 8, we will be defining what makes an 
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association between two variables statistically significant. But for now, suffice to say that 
they should be related to one another in a manner that is unlikely to be explained by 
chance alone. The independent and dependent variables should co-vary in some way 
with each other. Otherwise, one cannot be the cause of the other. 

2. Time order: Second, we need to establish the time order. Just because two variables co- 
vary with one another, we cannot be sure that the independent variable is the cause and 
the dependent variable is the effect if we do not establish time order. What this means is 
that we need to be sure that the independent variable occurred before the dependent vari- 
able, if we are to ascertain cause-effect relationships. 

3. Non-spurious relationships: The final criterion is that we need to confirm that the 
relationship between the independent and dependent variables was not a spurious rela- 
tionship caused by a third or extraneous variable. When two variables appear to be caus- 
ally related, but in reality, that relationship can be attributed to an external, confounding 
third variable, that is called a spurious relationship. For example, in the gaming example 
that we saw earlier, if we don’t control for an extraneous variable such as “level of vio- 
lence,’ then we might not be sure that the independent variable (exposure to positive 
portrayals of Black women) is the cause for the dependent or effect variable (level of 
racialized sexism). 


One useful way to remember these principles comes from a famous adage you may have 
heard: “Correlation does not mean causation.” Just because two variables are associ- 
ated with one another—meaning that a change in the values of one variable occurs in 
relationship with a change in the values of the other—that does not mean that we can 
claim a causal relationship between the two. Rather, there must be correlation (or associ- 
ation) in addition to an established time order and the ability to control for third variables. 

Beyond establishing the cause-effect relationship between the independent and depend- 
ent variables, researchers could also use experiments to understand the causal mechanism 
underlying them. A causal mechanism is any process that clarifies the relationship 
between the independent and dependent variables (Schutt, 2015). It helps to explain 
why the independent variable causes changes in the dependent variable. Sometimes the 
independent variable does not directly affect the dependent variable, but the independent 
variable affects another variable (called the intervening or mediating variable), which, in 
turn, affects the dependent variable. In other words, intervening or mediating variables 
(also known as mediators) are those variables that intervene between the independent 
and dependent variable, as an explanatory variable in the causal mechanism. For example, 
an experiment conducted by political communication scholars Matthes and Schmuck 
(2017) was interested in exploring the causal mechanism that explains the relationship 
between exposure to anti-immigrant political ads and attitudes toward immigrants. They 
found that this relationship was mediated by “intergroup anxiety.” That is, the mediating 
variable “intergroup anxiety” intervened between exposure to anti-immigrant ads and 
attitudes toward immigrants. Exposure to anti-immigrant ads increased intergroup anxi- 
ety for participants. Intergroup anxiety, in turn, led to increased negative attitudes toward 
immigrants. 

Apart from mediating variables, other types of variables that can help explain causal 
mechanisms are moderating variables. These variables determine the context in which a 
relationship between the independent and dependent variables holds good. Moderating 
variables or moderators are those that make the relationship between the independ- 
ent and dependent variables stronger or weaker. With the earlier example that we saw, 
the relationship between exposure to anti-immigrant ads and negative attitudes toward 
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Figure 6.3 Mediator and moderator variables 


Source: Emily Riewestahl 


immigrants was moderated by level of education. That is, the moderating variable “level 
of education” determined the context or condition under which the relationship was 
stronger or weaker. In other words, among participants with lower education, researchers 
observed that anti-immigrant political ads had stronger effects on negative attitudes toward 
immigrants. 

In Figure 6.3, X is the independent variable, which leads to M „ the mediating variable, 
which in turn, leads to Y variable, which is the outcome variable. The moderating vari- 
able, M „ is a contextual variable that influences both X and Y. Using the earlier exam- 
ple, the relationship between exposure to anti-immigrant ads (X) and attitudes toward 
immigrants (Y) is mediated by intergroup anxiety (M) and moderated by level of educa- 
tion, M.. 

As you can see in the previous examples, using independent variables, dependent vari- 
ables, control variables, mediating variables, and moderating variables, experiments can help 
social justice scholars start to understand the theoretical reasoning and causal mechanisms that 
explain the effects on dependent variables. For many social issues, knowing exactly what the 
cause is and the theoretical explanation for why it causes the effect is invaluable in making 
systematic long-term policy changes. Several research methods textbooks make it seem like 
it is impossible to establish causality due to confounding variables and other threats to valid- 
ity. In reality, even if causal relationships are not established with 100% certainty, for instance 
if not all possible confounding variables are accounted for, experiments can still be helpful 
in unraveling the theoretical mechanisms that underlie relationships among key variables of 
interest (Webster & Sell, 2014). Through a series of experiments, the researcher can start 
unpacking the theoretical mechanisms that explain how and why the cause variable affects 
the outcome variable. 


True Experiments 


There are many types of experiments such as true experiments, quasi-experiments, and pre- 
experiments. True experiments are typically conducted in lab-based settings under a high 
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degree of control. There are three requirements for an experiment to be considered a true 
experiment (Schutt, 2015): 


1. A true experiment should have at least two comparison groups. Comparison groups are 
those that are exposed to different values of the independent variable, thus being exposed 
to different treatments. In its simplest form, a true experiment is when participants are ran- 
domly assigned (see item 3 below for a definition) to one of two treatment groups—typi- 
cally, the experimental group (presence of the treatment) and the control group (absence of 
the treatment). However, not all comparison groups need to be experimental and control 
groups. They can also just be two different treatments. For example, let’s take an experiment 
that examines how positive or negative news portrayals of Muslims influence perceptions of 
this group. The independent variable is manipulated into two types of treatments: positive 
and negative news stories, rather than the presence or absence of the experimental condition. 

2. The temporal order (or time order) of the variables matters in experiments. The 
temporal order should be established so that it is clear that the independent variable was 
manipulated before the dependent variable was measured. In the earlier example, the 
researcher was interested in how exposure to positive versus negative news stories about 
Muslims influences attitudes toward this group. Unless we can be sure that the exposure 
to the news stories happened before the measurement of attitudes, we cannot be confi- 
dent that the independent variable causes the dependent variable. 

3. There has to be a random assignment to the two (or more) conditions. Random assign- 
ment means that a random process is used to decide which comparison group the par- 
ticipant will participate in. Random assignment is extremely important to make sure 
that every participant in the sample has an equal chance of being in a specific research 
condition. For example, in the experiment discussed earlier, it is the experimenter’s job 
to make sure that everyone in the sample has an equal chance of reading either the posi- 
tive or negative news story about Muslims. There should be no bias in deciding whether 
a particular participant will be assigned to the positive or the negative story condition. 
Only then can the researcher be confident that the participants in all the conditions are 
equivalent to each other and do not differ in any significant way along key characteristics. 
That is, to establish causation, the differences in the dependent variable should not be 
caused by differences in characteristics of the participants in the various conditions but 
should be clearly attributable only to the independent variable. Later in this chapter, we 
will describe how exactly a researcher assigns participants to various conditions. 


Given these requirements, true experiments are the best way to test for causal relationships 
between the independent and dependent variables. A well-designed true experiment offers a high 
degree of control over the independent variables, which helps to observe their effects on depend- 
ent variables in a controlled research setting. It isolates the key factors by controlling for extraneous 
variables so that the researcher can be confident that the changes in the dependent variables are 
associated only with the changes in the independent variables. For example, in another study, the 
second author of this book was interested in finding out whether exposure to media literacy train- 
ing reduces implicit racial bias activated by stereotypical news stories (Ramasubramanian, 2007). 
The independent variable is exposure to the media literacy video or control video. Exposure to 
the media literacy video should occur before implicit racial bias is measured to be sure of the time 
order. The researcher should randomly assign participants to the two conditions. 
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Figure 6.4 An example of key aspects of experimental design 


Source: Emily Riewestahl 


The simplest design is where there is only one independent variable, with two conditions. 
For example, in the earlier example, media literacy and control video were the two conditions 
of the independent variable. However, a single independent variable could be manipulated to 
have more than two levels as well. For example, researchers could manipulate the level of sex- 
ist humor in internet memes to be low, medium, or high. There may even be more than one 
independent variable in an experiment. Typically, anywhere from one to three independent 
variables are manipulated in a given experiment. 

When there is more than one independent variable, an experiment is called a facto- 
rial experiment. The more independent variables that are manipulated or measured in an 
experiment, the more complex and cumbersome the design becomes. A factorial design 
statement uses numbers and a cross-tabulation to visually depict the number of independ- 
ent variables and the levels of each independent variable. For example, let’s say a social justice 
researcher is interested in understanding how music videos influence sexist attitudes. She 
first identifies two independent variables of interest: (1) the level of sexual objectification in 
the video (high, medium, low) and (2) the type of music video (hip-hop, country, rock, and 
pop) that could influence the dependent variable, which is participants’ rape myth acceptance 
scores. Therefore, she would conduct a 3x4 factorial design with 12 conditions, which looks 
like Figure 6.5. The “3” in the factorial design statement represents the three values for level 
of sexual objectification (high, medium, and low) and the “4” represents the four music types 
(hip-hop, country, rock, and pop). Factorial designs can certainly help researchers understand 
complex and nuanced phenomena, but they also need to be used with care. As you may recall 
from Chapter 4, the larger the number of conditions, the greater the sample size that is needed 
to examine the relationships among the variables. Often, too, more complex designs call for 
more complex statistical procedures to compare the different conditions and understand the 
relationships among variables. 

All true experiments have to involve posttest measures, where the variations in the depend- 
ent variable are measured after the experimental group has received the treatment. Yet, a true 
experiment could use a posttest-only control group design, a pre-/posttest control group 
design, or a Solomon four-group design. If the dependent variable is only measured after the 
treatment, it is called a posttest-only control group design. 
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Figure 6.5 A 3x4 factorial experimental design 
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Some true experiments could also include pretests in which the dependent variable is 
measured before the treatment is applied to the experimental group. Such an experiment is 
said to have a pretest/posttest contol group design. Typically, the pretest and posttest 
are identical, if not equivalent, but simply administered at different points in time—before 
and after the experimental group has received the treatment. Using pretest/posttest designs 
gives the experimenter greater confidence that the changes in the dependent variable 
were only due to the treatment. However, the limitation of this design is that exposure to 
the pretest might interfere with how participants react to the treatment and the posttest 
measures. In an example of this type of design, the first author of this book conducted 
a pretest/posttest media literacy intervention with sixth graders (Scharrer, 2006). First, a 
pretest questionnaire measured students’ pre-existing levels of critical thinking and com- 
prehension of key concepts. The experimental intervention consisted of five one-hour 
visits over a six-week period to discuss various aspects of media literacy. There was also a 
control group of students who did not receive any media literacy training. At the end of 
the program, the students completed a post-test questionnaire, which was identical to the 
one used for the pretest. 

Solomon four-group design is one that combines that posttest-only control group 
design with the pretest/posttest control group design (Babbie, 1989). In this design, there are 
two experimental groups and two control groups—one that includes only post-test meas- 
ures, and another that includes pretest/posttest measures. There are four groups in total, one 
pretest/posttest experimental group, a posttest-only experimental group, a pretest/posttest 
control group, and a posttest only control group. In their study on how exposure to admirable 
LGBTQ+ media characters decreases prejudice toward LGBTQ+ individuals, Schiappa and 
colleagues (2005) used a Solomon four-group design. Participants were randomly assigned to 
one of four groups: posttest only control, pretest/posttest control, posttest only treatment, and 
pretest/posttest treatment. The treatment groups saw three episodes of the TV show Queer 
Eye, which included positive LGBTQ+ portrayals. The control group was not exposed to 
any TV show. The pretest and posttest measures included the Attitudes Toward Lesbians and 
Gay Men (ATLG) measures. This design is especially helpful if researchers are concerned that 
administering the pretest might sensitize the participants and influence how to respond to the 
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treatment. The drawback of this design is that the researcher needs double the time, funds, 
sample size, and resources to conduct the study. 


Quasi-experiments 


Although true experiments are ideal, they are not always feasible. It could take up a lot of time 
and resources to conduct true experiments. Even if resources are available, sometimes, it may 
not be ethical or practically possible to either have a control group or use random assignment 
to conditions. For instance, it might not be possible to assign some students to a media literacy 
intervention within the same classroom and not offer the same program to others in the room. 
Other limitations of true experiments are that they often do not reflect real-world natural 
settings since they are mostly conducted in laboratory settings in highly controlled environ- 
ments that are quite artificial. This is where experimental designs such as quasi-experiments 
and pre-experiments can be helpful. 

Quasi-Experiments are different from true experiments in that they typically lack random 
assignment to experimental and control conditions because of practical or ethical constraints 
(Mark & Reichardt, 2013; Wrench, Thomas-Maddox, Richmond, & McCroskey, 2012). 
This means that quasi-experiments compare the experimental group with a nonequivalent 
comparison group. Although quasi-experiments do not allow for random assignment, they 
are particularly helpful in understanding how programs and interventions work in the real 
world. For example, the researcher might compare two communities that live nearby, one that 
participates in a program on HIV/AIDS prevention through safe sex practices and another 
that does not. The community members were not randomly assigned to the intervention or 
control conditions since they were already living in those areas even before the research study 
started. This is a quasi-experiment because there is no randomization. Quasi-experiments can 
use either posttest only design or pretest/posttest design. But their defining feature is that they 
involve a comparison group that is not randomly assigned, which makes it a nonequivalent 
comparison group (Mark & Reichardt, 2013). The lack of random assignment means that 
the participants in the experimental group and control group could differ across some other 
dimensions other than exposure to the experimental treatment. Therefore, claims of causation 
in quasi-experiments are usually made more cautiously. 


Pre-Experiments 


Occasionally, neither quasi-experiments nor true experiments are possible. In these circum- 
stances, researchers might conduct pre-experiments. Pre-experimental designs are also 
sometimes used as a first step before conducting a true experiment or quasi-experiment. In 
pre-experimental designs, typically there is no random assignment and there is also no control 
group that is used as a basis of comparison. Although not ideal, pre-experiments are helpful 
when the sample size is not large enough to split into experimental and control groups or if 
the researcher does not have access to a control group. 

The most common type of pre-experiment is a one-group pretest posttest pre-exper- 
iment where there is no control group but the dependent variable is measured before and 
after the treatment. For instance, a Communication researcher might be interested in testing 
whether empathy levels in the participants change significantly before and after participating 
in a two-hour workshop on race relations. In this example, the researcher is able to compare 
the pretest and posttest measures of empathy before and after the program, but the lack of a 
control group means that it becomes harder to rule out confounding variables. 
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Another type of pre-experiment is one that is simply a one-shot case study without a 
control group or pretest/posttest measures. For example, a researcher interested in studying 
social media use during natural disasters might simply collect such data in the affected com- 
munities using posttest measures since they are not able to employ a control group or do 
pre-/posttest comparisons. For all types of pre-experiments, claims of causal relationships are 
tenuous, since the criteria for causation we discussed earlier in the chapter have not been met. 


Field or Natural Experiments 


Experiments conducted in natural real-world settings are called field or natural experi- 
ments (Carpenter, Harrison, & List, 2005; Driskell, King, & Driskell, 2014). Field experi- 
ments are especially helpful for social justice researchers interested in understanding real-life 
behaviors, outside the confines of laboratories, in more naturalistic settings. Field experi- 
ments are commonly used to evaluate the effectiveness of programs designed to solve practical 
social issues in real-world settings where the complex nuances of lived realities can be exam- 
ined. They are conducted within the natural settings of participants’ everyday lives in terms 
of behaviors, contexts, and topics. These are environments in which participants naturally 
undertake the tasks and behaviors that the researcher is interested in studying. Therefore, it 
is more likely that the findings apply and translate into practical actionable solutions to social 
issues. 

Field experiments are also referred to as interventions or program research. Interventions, 
by definition, are programs and strategies purposely designed to create change. Interven- 
tions could be conducted over short or long periods of time. For example, a year-long field 
experiment conducted in Rwanda by Elizabeth Paluck (2009) helped researchers understand 
the role of radio soap operas in reducing inter-ethnic prejudice by shifting social norms 
and increasing empathy. The communities were randomly assigned to the treatment condi- 
tion (the radio show about reconciliation and prejudice reduction) or the control condition 
(another radio show on a health topic). The communities were matched as closely as possible 
in terms of population characteristics and size. The effects of the intervention on shifting per- 
sonal beliefs, social norms, intergroup attitudes, and behaviors were collected and analyzed. 

Whereas laboratory studies typically measure short-term immediate effects, interventions 
in the field often last longer. Recall from Chapter 5 that cross-sectional designs involve meas- 
uring the independent and dependent variables at one point of time, whereas longitudinal 
designs involve multiple points of measurement. Traditionally, true experiments tend to be 
cross-sectional, where data are collected in a single-session lab setting. In contrast, when a 
researcher conducts field experiments, it is quite common to assess interventions using a 
longitudinal design that involves multiple measurements taken during the course of the inter- 
vention. This type of longitudinal experimental design is called a time series experimental 
design. The advantage of this design is that the researcher can understand how the outcome 
variables unfold over time. 

The biggest advantage of field experiments is that they help researchers observe partici- 
pants in naturally occurring environments. They have high external validity since they simu- 
late natural settings. The drawback of field experiments, though, is that the researcher has 
very little control over the environment or other factors (Parvanta & Bass, 2018). Therefore, 
the experiments are often influenced by other external factors such as geographic location, 
political context, and other variables out of the control of the researcher. They can also be 
expensive and time-consuming to conduct. Another concern with field experiments is that 
participants typically do not know that they are a part of an experiment. In the absence of 
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full consent, researchers try to minimize risks as much as possible. Despite these limitations, 
field experiments are useful for social justice scholars, especially in community-based research. 


Using Experiments for Social Justice Research 


We have seen a variety of experimental designs in the previous sections: lab versus field exper- 
iments, true versus quasi- and pre-experiments, posttest only versus pretest/posttest designs, 
and so on. There are several trade-offs between different types of experiments, depending on 
the topic, participants, and resources available to the researchers. So, social justice researchers 
should carefully consider which type of design is best for their goals and purposes. 

Social justice researchers are interested in using experimental research to systematically 
counter inequalities. They could do that in at least two different ways. They could use lab- 
based experiments to first study social justice issues to understand cause-effect relationships 
within controlled settings. Once the key cause and effect variables have been determined, 
they might then move to test the outcomes of specific interventions using a field experiment 
(Driskell et al., 2014). Indeed, the opportunity to study outcomes associated with interven- 
tions is an important advantage of experimental design for the social justice researcher. 

Experimental methodology, with its unique claim toward cause and effect relationships, is 
particularly well suited to study change. Communication researchers might be interested in 
designing and testing interventions such as dialogues around race and racism in the classroom 
to work toward an inclusive campus, communication campaigns to address health inequalities, 
organizational workshops to promote an inclusive climate, or media literacy initiatives to spur 
media activism in favor of diverse representations, just to name a few. The experiments might 
take place in the field or in the lab, but in either case, the experimental design can shed light 
on whether and under what conditions interventions produce their intended change. 

For lab-based studies, researchers should be mindful of the time and effort it would take 
participants to participate in the study. Researchers are often focused purely on their research 
agenda, ensuring that the experiment is designed with tight controls, and perhaps even simply 
mechanically adhering to the minimal requirements of IRBs. They sometimes are not aware 
of or do not pay attention to the challenges, risks, and discomforts that participants might 
have to go through to participate in an experiment. In conceptualizing their research partici- 
pants merely as “subjects” who are research materials, they might fail to recognize the human- 
ity of their participants (Hegtvedt, 2014). Failing to incorporate participants’ perspectives and 
preferences is especially problematic and even harmful for researchers working on social issues 
with aggrieved populations. Researchers should understand that participants are volunteers. 
Those volunteers should not feel compelled or obligated to stay in the study, especially if they 
experience discomfort or risks. 

Researchers need to think through the study procedures, the timing, transportation, acces- 
sibility of spaces, and compensation as they recruit their participants to make it easy and 
convenient for participants to participate in the study. Keeping the participants’ convenience 
in mind, the researchers should ensure that the experiment is conducted in spaces that are 
accessible. They could offer help with compensating for public transportation, a meal, or 
other incentives to encourage and support participants, whenever possible. Populations that 
social justice scholars are working with may be working on multiple jobs or have constraints 
that prevent them from being able to set aside a lot of time to attend a research experiment, 
even if they wanted to. Therefore, researchers should do their best to offer multiple time slots 
and opportunities for the participants to be part of the study. 

Within social justice work, applied experimental work would likely involve working within 
and with marginalized communities and may even occur over an extended period of time. 
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Developing a positive relationship with research participants in longer term projects helps 
create a reciprocal relationship of care and trust. Such an approach requires open communi- 
cation, active listening, and skillful planning. For example, Bekalu and Eggermont (2014) 
conducted a field experiment in Northwest Ethiopia to study the effects of various types of 
HIV/AIDS prevention campaign messages. The researchers translated the stimulus materials 
to the first language of the participants, Amharic, and also made sure to account for language 
and cultural differences between urban and rural residents. Their research was able to find that 
residents of this area responded more favorably to a health message about HIV that incorpo- 
rated local language and culture. It is easy to see from this example of a study of an interven- 
tion, that the findings have important implications for health and well-being. 

While conducting experiments for social justice, it is especially crucial to ensure equal 
distribution of benefits across participants (Schutt, 2015). Given that the goal of social justice 
work is to reduce social inequalities, the researcher is faced with a difficult decision about 
the ethics of selective distribution of the benefits of an intervention on one community or a 
section of the population rather than be received by all participants. For example, if a Com- 
munication researcher conducts a series of workshops on interviewing skills for those in the 
experimental group while the control group participants do not receive this information, it is 
possible that the inequalities in job opportunities and performance are further exacerbated by 
the experiment. In this instance, the researcher could perhaps offer similar workshops to the 
control group after the experiment is completed or offer to provide them in a different format 
such as webinars to ensure that the control group members are not deprived of the benefits 
that the experimental group receives. 


Scholar Spotlight: Dr. Brian Southwell 


Brian Southwell is Director of the Science in the Public Sphere Program in the Center 
for Communication Science at RTI International, an adjunct professor with Duke Uni- 
versity, and a graduate faculty member (in Media and Journalism) and Adjunct Associate 
Professor (of Health Behavior) at the University of North Carolina at Chapel Hill. Dr. 
Southwell is an expert on health communication, public health, science communica- 
tion, and public scholarship. His work has examined topics such as smoking cessation, 
drug use, heart attack prevention, HIV/AIDS, and water conservation. Dr. Southwell’s 
research contributions appear in more than 100 journal articles and chapters in his vari- 
ous books, including his latest one on Misinformation and Mass Audiences (2018). South- 
well also is a host of a public radio show, The Measure of Everyday Life, for WNCU. An 
excerpt from an interview with Dr. Southwell follows. 


Q: Do you think quantitative methods in general and experiments, in particular, can be tools 
to work toward social justice? Why or why not? 

Dr. Southwell: Yes, quantitative methods, like qualitative methods, can answer rel- 
evant research questions about social justice. The key to useful social science lies 
in the creativity and relevance of the questions we ask, and at least some questions 
relevant to social justice can be addressed with quantitative research. Research on 
social justice should consider both individual-level and community-level variables. 
We need to understand what people think and how they behave and how they 
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Figure 6.6 Scholar Spotlight: Dr. Brian Southwell 


interact with their information environments to arrive at such perceptions and 
behavior. Quantitative data can be useful in measuring perceptions and behavior 
and can describe patterns of information. A range of study designs, including 
experiments, can help us understand the effects of particular stimuli, effects that 
can point to promising interventions, or explain concerning patterns of existing 
behavior. 

Q: What motivates you to do quantitative research in general? What types of issues, topics, or 
interventions have you examined using experimental methods, in particular? 

Dr. Southwell: As an undergraduate, I had a sense that our understanding of many 
aspects of human life should not be quantified. That was an overly simplistic stance, 
I now realize. None of our measures offer perfect replication of the phenomenon 
measured, but that does not mean there cannot be useful correspondence between 
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our measures and underlying phenomenon. Once you realize there is room for 
artistry and creativity in the use of quantitative measures for the purpose of pre- 
dicting future human interaction, you can see quantitative research as a poten- 
tially, but not inherently, useful tool. An example of this possibility les in the 
discrepancy between claims people make about the influence of mediated images 
and the actual patterns of everyday life we know people experience, patterns that 
include all sorts of conversations and exposures to information. Certainly, widely 
broadcast advertising content can have effects, but we also need to understand 
what roles social network interactions and other factors can play in predicting 
the influence of various campaign efforts. By measuring and modeling a range 
of influences, including both macro- and micro-level forces, we can get a clearer 
picture of what happens when a well-intentioned public service announcement is 
developed and released into the world for the purpose of eliciting social change. 

Q: What challenges arise in your research, especially in terms of designing and evaluating the 
effectiveness of health-related interventions? What are some ways you have overcome them? 

Dr. Southwell: We continue to face challenges in measuring even basic concepts 
related to human communication. In some ways, our tendency to pay attention 
to newly salient buzzwords and phrases in popular discourse rather than to care- 
fully ensure adequate measurement of mundane concepts has limited our work. 
Some of my early work on campaign exposure led me to realize that much of 
our reporting on exposure really reflected the dynamics of human memory in 
interaction with information environments, meaning we needed to do multilevel 
modeling work, e.g., Southwell (2005). Decades ago, we rushed ahead to do work 
on interactivity, for example, when we often did not adequately define what we 
meant by that. More recently, we are seeing that happen as people have started 
to do work on misinformation before we really have defined the key variables 
involved. (See Southwell, Thorson, & Sheble, 2018, for discussion.) 

Q: What are your experiences and/or your hopes regarding how your research can be used to 
bring about social change? 

Dr. Southwell: We need to understand how information flows in society in order 
to best support people as they make decisions about their own behavior and as 
they interact with other people in various communities. We need to ensure that 
people have access to the information they need and that our collective conversa- 
tions include many different voices. Rigorous research that helps us understand 
how people make sense of science, health, medicine, policymaking, or even civic 
decision making could be a basis for our efforts to improve the human condition. 
Social change without rigorous, peer-reviewed research could devolve into shout- 
ing matches and empty slogans. By investigating how humans interact with their 
information environments, we can find ways to respectfully address all people to 
invite them to improve life for all of us. 


Designing and Conducting an Experimental Study 


Conducting experiments, be they laboratory-based studies or field experiments, involves 
multiple steps and stages of planning. It can get messy and is an involved process. One has 
to spend a lot of thought and time in identifying which samples to recruit, how to manipu- 
late the variables, how to control for extraneous factors, and how to measure the dependent 


140 Studying Responses 


variables. When designing applied field experiments, it is advisable to work closely with key 
stakeholders from the community by involving them all the way from initial design to dis- 
semination of the findings, as need be. Next, we discuss various practical and methodological 
considerations to keep in mind as you design and conduct experiments. 


Recruiting Participants 


For many experiments, especially lab-based ones at universities, researchers typically recruit 
student samples as a standard convenience sample. As we have seen in Chapter 4 on sam- 
pling, although convenience samples are nonprobabilistic and cannot generalize to a larger 
population, they are still useful for researchers. Over a series of experiments, for instance, 
researchers could nuance their theories through testing various hypotheses using conveni- 
ence samples of students. Indeed, many academic departments, especially within Psychology 
and increasingly within Communication, at research institutions have established partici- 
pant pools of students who sign up for various studies through the course of the semester at 
the departmental-level rather than being recruited by individual researchers. Depending on 
the variables that are being examined by a social justice researcher, college students might 
not be a relevant or useful population to test certain topics. This is especially important 
to consider for social justice work where the communities of interest have different kinds 
of skill sets, knowledge, and familiarity with the experimental tasks as compared to, say, 
convenience samples of college students. While working with other non-student samples, 
researchers could use flyers, posters, newspaper ads, and community newsletters to reach 
out to these populations. 

It is important to try to balance the number of participants across various conditions. In a 
simple experiment with two conditions (e.g., experimental group and control group), if we 
recruit 30 participants per condition, this means at least 60 participants. Considering that not 
everyone who signs up for a study might actually show up or complete it, it is always bet- 
ter to recruit more participants than the required sample size. See Chapter 4 on what would 
be considered a representative and large enough sample. As more conditions are added to 
an experiment, such as in a factorial experiment, the sample size increases proportionately. 
For example, if it is a 3x4 factorial experiment, the number of conditions is 3x4 = 12. If the 
researchers would like 30 participants per condition, the researchers need to recruit a bare 
minimum of 12 x 30 = 360 participants, which could be expensive and time consuming. 


Obtaining Consent 


Once the researchers have decided on their sample population, one of the first steps involved 
in designing and administering an experiment is obtaining permission from IRB to conduct 
the study in an ethical manner. We have seen in Chapter 2 the detailed steps involved in 
conducting ethically appropriate studies. Given that most of the experiments within Com- 
munication use human participants, it is important to consider how their rights, dignity, and 
preferences are honored from the beginning to the end of their experience. Since experi- 
ments involve some type of intervention or exposure to a treatment, it is especially important 
that human subjects protections are in place. By means of comparison, surveys, for instance, 
typically ask participants to report on their opinions, attitudes, past or planned behaviors, etc. 
But experiments, by definition, manipulate something in participants’ experiences and there- 
fore require particular care and conscientiousness. 

Deception is an ethical consideration with which social justice researchers using exper- 
imental methods sometimes have to contend. For the sake of experimental manipulation 
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and the purposes of the research study, occasionally, researchers might have to misrepresent 
information or mislead participants. For example, one of the research team members might 
have to play the role of a fake participant in order for the manipulation to be effective. 
Such members are called confederates (we will learn more about them later in this chap- 
ter). Outright deception is quite rare in experimental research within Communication, 
and we believe it should be avoided whenever possible since it can be ethically murky. But 
sometimes, lack of transparency is necessary for research designs. Social justice researchers 
need to weigh the costs of temporarily keeping participants uninformed about the goals of 
an experiment against the benefits of the knowledge that can be gained (Hegtvedt, 2014; 
Schutt, 2015). In any case, all researchers have the obligation to inform participants if and 
how they were misled after the data has been collected. That information is usually pro- 
vided in what is called a debriefing, which involves explaining the research design and 
purpose in writing or verbally at the close of the study. We will discuss debriefing more at 
the end of this chapter. 

Participants should be informed ahead of time about the possible risks and benefits of the 
study. They should only participate voluntarily without any type of coercion and should be 
allowed to leave the study without any penalties if they experience any type of discomfort 
or stress. Unlike other types of research such as that conducted in the biomedical field, 
typical Communication studies are more likely to run the risk of causing psychological 
or social distress rather than physical harm. Social justice scholars, in particular, are likely 
to bring up challenging and sensitive topics such as discrimination, violence, incivility, 
exclusion, abuse, and inequalities, which could cause emotional and psychological stress 
among participants in revealing and discussing. Therefore, the participants have the right 
to be informed of these stressors and discomforts that might be above those routinely 
experienced in the everyday course of life for these participants. It is also possible that the 
lived reality of stigmatized and marginalized groups is already fraught with similar risks 
and stressors in their everyday lives. When working with children, refugees, prisoners, or 
other vulnerable groups, researchers have an ethical obligation to ensure that they are not 
mistreated nor retraumatized. 


Manipulating the Independent Variable 


As we have seen, manipulation means that the researcher deliberately or purposefully alters 
or varies some aspect of the independent variable. This could mean determining how, when, 
where, and by how much to alter the independent variable. Researchers can manipulate 
information provided to participants about the task they are asked to perform in the study, 
the people involved in it, or the participants themselves. The purpose is to observe how these 
intentional changes to the independent variable affect the dependent variable. 

Within Communication research, independent variables can be manipulated using 
stimulus materials, role-play scenarios, and confederates (Wrench et al., 2012). Expos- 
ing participants to some type of stimulus material is one of the most common types of 
manipulations in Communication studies. For example, participants in the experimental 
condition could be asked to watch a television program on LGBTQ+ issues or browse 
through an online campaign for a political candidate or play a violent video game. In some 
cases, this stimulus material could be as simple as manipulating a written prompt that gives 
some sort of instruction for participants. For instance, participants in one group could 
be asked to recall a happy time in their life or a sad time in their life. In other cases, it 
could involve multimedia materials such as videos, films, audio recordings, or web-based 
materials. 
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Figure 6.7 A field experiment in a classroom setting 


Source: Cpl. Khalil Ross 


Another type of manipulation of the independent variable could involve the participants 
doing different types of communication activities such as role-play scenarios. In such stud- 
ies, the participants are actively involved in some task that involves more than simply viewing 
a stimulus. For example, participants could be asked to simulate how to respond during a 
natural disaster scenario using virtual reality or a video game. For example, researchers used 
role-play scenarios through avatars in a video game to understand implicit racial bias (Peck, 
Seinfeld, Aglioti, & Slater, 2013). White participants were asked by researchers to embody 
different skin tones in their avatars to examine whether playing an avatar of a different skin 
color influences their racial attitudes. Racial attitudes were measured before and after the role 
play scenario. Those who embodied dark skinned bodies in the role play scenario demon- 
strated reduced racial biases. 

It is also possible to manipulate independent variables using confederates. Confederates 
are those who are part of the experiment, but the participants are not aware of this. For 
instance, confederates could be recruited by the researchers to play the role of a fake par- 
ticipant or to serve as a bystander to make the manipulation look realistic. The use of con- 
federates does introduce some level of deception to the experimental design. For example, 
confederates could be placed within a video game study to either assist with or thwart the 
participant as their partner in a multiplayer gaming scenario. The participant is made to 
believe that the confederate is another participant in the study, but in reality they are part 
of the research team, and what they do represents different levels of the independent vari- 
able. Although confederates are typically employed in lab-based studies, they could also be 
used in field experiments. In a field experiment by Katherine Haenschen (2016) on political 
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communication, for example, confederates were recruited to publicize participants’ prior vot- 
ing or non-voting behaviors by tagging them on Facebook updates about their voting behav- 
iors. The study found that doing so increased voter turnout of participants much more than 
face-to-face methods used for encouraging political participation. 

Regardless of how the independent variable is manipulated, one issue that is often of con- 
cern to researchers is strength of manipulation. That is, whether the manipulation is strong 
or substantial enough to bring about changes in the dependent variable. As Walker (2014, 
p. 131) states: “Subtlety is out of place in experimental design.” They recommend ensur- 
ing that participants get the information about the manipulation at least three times during 
the experiment so that, even if they are not attentive, crucial bits of information about the 
manipulation are still comprehended the second or third time. While this is not always pos- 
sible to expose participants to the stimuli multiple times, researchers should at least make sure 
that the various treatment conditions are distinctly and clearly different from one another. If 
the manipulation is too subtle, some participants might fail to notice and therefore, even if 
they are assigned to a certain comparison group, the participants might not really be in that 
group. For example, if you are manipulating the level of humor of a presentation to be low, 
medium, or high to examine its effects on audience recall, you should ensure that the high 
humor condition is substantially funnier than the medium condition, just as the medium one 
is funnier than the low humor condition. 

Sometimes the effects of the independent variable on the dependent variable might only 
manifest itself when the manipulation is quite strong. This phenomenon is referred to as 
threshold effects (Kuipers & Hysom, 2014). Unless the independent variable is above a 
certain threshold, the dependent variable does not change significantly. For instance, in the 
earlier example, it is possible that low and medium doses of humor do not have any impact 
on audience recall. If the researchers had not manipulated the independent variable to also 
include a high level of humor, they could have missed accounting for the relationship between 
the independent and dependent variables. 

Since topics of interest to social justice scholars such as racism, discrimination, and vio- 
lence could be multifaceted, with different meanings for various groups, it is important to 
ensure that the way in which the researcher interprets these concepts is similar to how the 
target populations also understand them. It is always a good idea to pretest the manipula- 
tions of the independent variables. For instance, in an experiment conducted by Martinez 
and Ramasubramanian (2015), the researchers were interested in studying if exposure to 
race-based comedy will lead to participants rating an alleged Latino offender more culpable 
than a White one. They conducted pretests to test the effectiveness of the stimuli. The first 
pretest was to ensure that the names of the alleged offenders were seen as representing the 
racial groups (Latinx and Whites, respectively). The second and third pretest focused on 
making sure that the comedic material used in the experiment was appropriate, not some- 
thing that the participants were already familiar with, and was seen as race-based humor, 
as intended. 

Researchers also do a manipulation check, which is a secondary evaluation within the 
experiment to ensure that the manipulation worked as intended by the researchers (Wrench et 
al, 2012). For instance, apart from the dependent variable, the experiment on sexual objec- 
tification in music videos that we used as an earlier example, could also include a series of 
questions within the experiment itself asking the participants to rate how sexual the stimulus 
material is. In the experiment described earlier on race-based comedy, the researchers also 
conducted a manipulation check within the actual experiment by including questions about 
the race of the alleged offenders and how funny the race-based comedic segments were. 
In both of these examples, we see the importance of manipulation checks. Rather than 
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assuming that the participants shared the researcher’s sense of the stimulus materials, when the 
researcher includes manipulation check items in the study, they now have evidence for that. 


Randomly Assigning Participants to Conditions 


Random assignment means that there should be no systematic bias (such as participant bias or 
experimenter bias) in assigning participants to conditions (Schutt, 2015). Random assignment 
is important for experiments because of the third criterion for causation that we outlined 
earlier in this chapter. This is the requirement that the relationship between the independent 
variable (the predictor) and the dependent variable (the outcome) cannot be explained away 
by some other factor. If there is no random assignment to conditions, there may be a com- 
peting explanation for a relationship found between the independent variable and dependent 
variable due to the fact that the groups formed across conditions were not equivalent. If there 
is random assignment to conditions, we assume any additional factors associated with the 
people in the conditions are spread out across all conditions rather than concentrated in one. 
For example, a participant who wants to be assigned to the same condition as their friend is 
violating the random assignment requirement of experiments. Similarly, if a researcher decides 
that a participant appears to fit better in the experimental rather than control group, then 
they are bringing in their own prejudices and biases into the assignment to conditions, which 
means that the assignment to conditions is no longer random. 

There are a number of ways to assign participants randomly to various comparison groups 
in an experiment. One could simply flip a coin so that if it shows heads, they land in the 
experimental group; if it shows tails, they are assigned to the control group, or vice-versa. 
One could also draw their condition from a hat. For instance, if participants have to be 
assigned to a high, medium, or low level of humor, then each of these conditions could be 
assigned a number or letter. Equal numbers of pieces of papers with those letters (e.g., A, B, 
C, or 1, 2, 3) are placed in the hat from which the participant pulls out a piece of paper that 
determines the condition to which condition they are assigned. One could also use a random 
generator table or online random generator site such as www.randomizer.org. If you need 30 
participants for each of three conditions, you could pick the first 30 numbers at random from 
1 to 90 for condition one, the next 30 for condition two, and the remaining will be in condi- 
tion three. If the experiment involves asking participants to provide data through a computer, 
you can also use computer-based software to randomly assign each participant at a computer 
to one of the conditions at random. 

Random assignment assists greatly in the experimenter’ ability to make a causal claim, 
but it is not a perfect solution. In many places, because there is a high degree of correlation 
between demographics such as race, education, and family income with geographic location, 
even if participants are randomly selected, their demographic differences are highly correlated 
with their geographic location. Therefore, it is important to collect demographic informa- 
tion regardless of random assignment to make sure that the participants in various conditions 
do not differ significantly across key factors. Indeed, it is standard practice when conducting 
an experiment not only to manipulate the independent variable and measure any dependent 
variable of interest but also to measure demographic variables and/or any other control vari- 
ables that may be relevant. Doing so allows the researcher to know, statistically, how taking 
those additional variables into account impacts the relationship between the independent and 
the dependent variable(s). We further discuss controlling for extraneous variables in the text 
that follows. 

When conducting field experiments, researchers often cannot randomize participants into 
various treatment conditions due to practical and/or ethical reasons. In this case, they would 
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Figure 6.8 Matched pair technique combined with random assignment, in a context in which two gender 
identities emerged in the sample 


Source: Emily Riewestahl 


likely use a quasi-experimental design without random assignment to conditions. When ran- 
dom assignment is not possible, matched pairing is another alternative, although not as ideal 
as random assignment (Figure 6.8). In the matching procedure, participants across various 
comparison groups are equated along key defining characteristics (Schutt, 2015). For exam- 
ple, individual matching could be done by matching up pairs of people in the experimental 
and control groups in terms of the same age, gender, race, or other individual characteristics. 
Occasionally, researchers might combine the matched pair and random assignment procedures 
by first matching the pairs and then randomly assigning pairs to various research conditions. 


Controlling for Extraneous Variables 


As social justice researchers in Communication, we need to be conscious of how we apply 
the idea of controlling for extraneous factors to experiments involving human participants. 
The researcher should try their best to anticipate all extraneous variables that could influence 
the outcome of the experiment beyond just the independent variable. As we have seen in the 
beginning of this chapter, confounding variables are problematic because they could lead to 
spurious relationships that provide alternative explanations for the association between the 
independent and dependent variables. 

One concern is Hawthorne effect, where merely being aware of being a participant in 
a study could make the participants change their behaviors (Schutt, 2015). For example, if 
participants know that they are being observed, they could behave in socially desirable ways 
to appear to be more productive, less violent, kinder, and so on. Other factors that need to be 
controlled in experiments pertain to the experimenters themselves. When the changes in the 
dependent variable are unintentionally influenced by the experimenter, this is called experi- 
menter effects (Wrench et al., 2012). For example, if the experimenter is more welcom- 
ing to the participants during the experimental condition rather than the control condition, 
we will not know if the changes in the dependent variable are caused by the independent 
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variable or due to the experimenter’s differing level of warmth toward the participants. To 
avoid experimenter effects, researchers often use double-blind experimental designs so that 
neither the researcher nor the participants are aware of the condition to which they have been 
assigned. 


Measuring the Dependent Variable 


Dependent variables are always measured in experiments, and the researcher has many options 
for how to measure them. One way in which dependent variables are measured are self- 
reports, in which the individual fills out their responses to items using a paper-and-pen 
or computer-based questionnaire. In Martinez and Ramasubramanian (2015), for example, 
participants’ ratings of the level of guilt of the alleged offender is a self-reported variable on a 
questionnaire. 

Another way in which one could measure the dependent variable is through observa- 
tional data. In the example of violent versus nonviolent video games, the level of aggression 
could be observed by the researchers using such indicators as the use of violent language, 
clenched fists, physically aggressive behaviors toward others, and so on. Note that in this 
case, the participant is not self-reporting their level of aggression. Instead, the researchers 
are observing whether or not the participant is behaving aggressively. In the Bushman and 
Anderson (2009) study, the researchers observed how many minutes it took the participants 
in a violent versus nonviolent media condition to help someone in distress and pain. This is 
an observational measure since the participants did not self-report their helpfulness, but this 
was recorded by the researchers. 


Figure 6.9 Biophysiological measures monitored in a lab experiment 


Source: Creative Commons 
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Another type of observational measure is when the researcher hooks up the participants 
to physiological measurement systems to monitor changes to biophysiological indica- 
tors such as heart rate, eye movements, skin conductance, or blood pressure. For example, 
Sukalla and colleagues (2016) manipulated health-related narratives from medical dramas to 
be emotional or not and then collected psychophysiological measures such as heart rate, skin 
conductance, and facial EMG using the Biopac system to examine participants’ arousal and 
attention levels as they watched the video clips. 


Debriefing 


Debriefing is an important aspect of experiments that, unfortunately, does not always get suf- 
ficient attention from researchers. It is the final aspect of the experiment where the researcher 
sets aside some time to discuss the goals and purposes of the experiment and allows for partici- 
pants to ask questions. This could be done both for lab and field experiments to communicate 
about the study to participants as well as gain insights and relevant feedback from them. If the 
experiment is done online, participants can be invited to contact the researcher with ques- 
tions or for a report of the purpose of the study. Debriefing can also be helpful to find out 
whether participants changed their responses to fit what they thought the researchers were 
trying to study. This is called demand characteristics, where participants try to answer in 
ways that would please the researcher and confirm what they think they are testing. It is espe- 
cially essential to include debriefing if deception is involved in the study. In fact, it is consid- 
ered unethical to not inform the participants about deception during debriefing. Sometimes 
experiments could make participants feel uncomfortable or stressed. The debriefing session 
allows researchers to monitor, acknowledge, and alleviate such outcomes by checking in with 
participants about ways to support them. For instance, if a study is about mental health, the 
experimenter could include contact information for counseling services as part of the debrief 
so that participants who are interested in pursuing counseling have that information at their 
disposal. 


Research in Action 


In this section, we describe a quasi-experimental intervention published in the Journal 
of Health Communication. Saul Johnson and colleagues (2018) were interested in study- 
ing the long-term impact of a school-based public health intervention in South Africa. 
Previous interventions relating to HIV prevention among adolescents had used either 
media platforms to communicate their message or school-based interventions to pro- 
mote better sexual health. This study combined both these approaches by examining 
how being part of a school-based extracurricular program that focused on emulating 
what they saw in a television program on HIV prevention mobilized them to take better 
responsibility for their health. 

“Soul Buddyz” was a popular television drama series that aired from 2000 to 2011 in 
South Africa using “edutainment”—the combination of education and entertainment— 
to discuss HIV prevention. In 2003, several Soul Buddyz Clubs (SBCs) were started in 
schools to create a platform to discuss and promote adolescent health and well-being. 
Some students volunteered to join the clubs and others who were considered vulnerable 
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were invited to them by their schools. Over 8,000 clubs with more than 182,000 mem- 
bers were established over a period of time. The members of the clubs engaged in 
activities such as reading special materials, organizing events, and holding debates and 
competitions on not just sexual health but other related topics such as alcohol use. The 
objective of this study was to test what impact participation in SBCs had in the long 
term on its members. The independent variable was whether or not the participants 
were in the SBC program as an adolescent. The dependent or outcome variables were 
multiple measures relating to health and well-being. 

The researchers picked 170 ex-SBC members from two provinces who had been 
members from 2004 to 2008. Since random assignment to condition was not possible, 
they resorted to using the matched-pair technique to find equivalent persons from the 
same communities who would serve as controls. Although a nonequivalent control group 
existed, the lack of random assignment makes this a quasi-experiment. Since the study was 
conducted in the field, it is also a field experiment or a program evaluation study. 

Here is how the researchers went about recruiting members for their treatment and 
control conditions (p. 1046): 


The complexity of tracking and recruiting ex-Buddyz required a multipronged 
recruitment approach. Radio adverts directed ex-Buddyz to a closed Facebook 
group. Snowballing was then used to recruit additional ex-Buddyz to the group. 
We also obtained a database of 3,421 SBC facilitators from Gauteng and KwaZulu- 
Natal from the Soul City Institute. Study staff called each facilitator requesting a 
list of ex-Buddyz names and contact details, which we used to contact ex-Buddyz. 
Additional ex-Buddyz were recruited using snowballing. Ex-Buddyz were then 
contacted using WhatsApp and asked to share the communication with other ex- 
Buddyz they could remember. Ex-Buddyz were directed to the Facebook group 
at each stage. 

We identified which communities most ex-Buddyz we had recruited were 
originally from (and where they attended primary school). We conducted simple 
random sampling from households in these communities to recruit appropriate 
community controls. Eligibility criteria included being aged 18—24 years old, liv- 
ing in those communities during 2004—2008 and not having been an SBC member 
while at primary school. 


The questionnaire was translated into nine local languages. The fieldwork was con- 
ducted in two sites and participants were reimbursed for travel costs. Trained fieldwork- 
ers were used for completing interviewer-administered questionnaires using a tablet 
device. Whenever possible, the sex of the interviewer was matched with that of the 
participant. Participants completed questions relating to sexual behavior, HIV status, 
alcohol use, drug use, pregnancy status, demographics, community participation, and 
involvement in gender-based violence. The debriefing session involved providing a 
referral list of useful contacts including counseling and clinic information. An optional 
and anonymized HIV testing by trained nurses was also available. 

We encourage you to read the rest of this study to find out whether those in the treat- 
ment group and those in the control group in this posttest-only quasi-experiment con- 
ducted in the field differed significantly from one another with regard to the dependent 
variables. 
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In the context of experiments, internal validity refers to the extent to which the researchers can 
be confident that the changes in the dependent variable are caused by the independent vari- 
able, by ruling out confounding variables. The fewer the confounding variables in an experi- 
ment, the greater the internal validity of the study. There are several factors that could lessen 
the internal validity of the experiment. Some of the factors that could influence internal 
validity are history, maturation, testing, regression to the mean, selection threat, and attrition 
(Babbie, 1989). Many of these threats can apply to any sort of experiment, but they tend to 
be especially common in field experiments using longitudinal designs. 

The threat of history refers to the likelihood that some other external event other than 
the treatment occurred between the pretest and posttest measures that has influenced the 
outcome of the study. Imagine that when the researcher is studying the effects of a sexual vio- 
lence prevention campaign, there is a major sexual violence case in the news between when 
the pretest and posttest measures are taken. That event would affect the internal validity of 
the study. Pre-experiments that lack control groups are especially likely to miss accounting 
for history effects. 

Whereas history refers to external events outside of the control of the researchers, matu- 
ration is more about internal factors associated with the participants themselves. This is 
especially likely to happen if the intervention is conducted over a long period of time, within 
which the participants themselves experience physical or mental changes as they grow and 
age. For example, a media literacy intervention with middle-schoolers that spans an entire 
academic year needs to take into consideration that the children’s cognitive, emotional, and 
behavioral abilities could be different between the pretest and the posttest measurements. 

Testing refers to how completing pretest measures influences posttest measures. For exam- 
ple, taking an exam a second time could lead you to have higher scores just because you are 
now more familiar with the measures. To avoid testing threat, researchers may want to adopt 
the Solomon four-group design. Or, if they do not need to measure change in participants’ 
responses to items directly, they may want to use a posttest only design. Some advocate for the 
use of equivalent but not identical measures at the pretest and posttest to lessen testing effects, 
but doing so does open the researcher up to criticism that they are not measuring the exact 
same thing both times. 

Another measurement-related threat to internal validity is instrumentation, which refers 
to changes in the measures that occur simply from the instrument that is being used to test the 
outcomes rather than from the treatment. In this scenario, it looks like meaningful change has 
occurred due to the manipulation, but the way that change is measured might have influenced 
the result. In the case of social sciences, this could happen because an item on a questionnaire 
was changed or some other aspect of observation is different, such as the experimenter who 
records observations. 

Another threat to internal validity is regression to the mean, in which posttest scores 
tend to be much closer to average or normal compared to pretest scores. For example, it is 
possible that participants report unusually high levels of stress when they first find out that 
they might be genetically inclined to get cancer. However, over time, even without any treat- 
ment or intervention, their stress levels might attenuate or regress to the mean. 

Other types of threats to internal validity relate to participants. Selection threat refers to 
the fact that participants selected to participate in the study may differ along some key char- 
acteristics that could affect the findings. For example, college students in a participant pool 
who participate routinely in multiple studies every semester only for extra credit might be less 
sincere in participating in the experimental task and completing questionnaires as compared 
to others. 
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Another threat to validity is attrition, which refers to the number of participants who 
have left the study partway through, without completing it. This is especially a concern in 
long-term studies that might involve multiple stages and a series of measures of the dependent 
variable over time. It is especially problematic if the participants who have dropped out of the 
study differ from those who have remained in the study in some relevant manner. As social 
justice scholars, if, for instance, most of the participants who drop out of a study are from 
lower economic classes or have mobility issues or need child care assistance, it shows that the 
study needs to be designed to better accommodate these types of participants. 


Increasing the Generalizability of Experiments 


One of the big concerns with experiments is the lack of generalizability of the findings of 
the experiment to real-world settings. Researchers need to ask themselves whether the results 
of their study conducted with the sample could be generalizable to other contexts, cultural 
groups, geographical locations, and time periods. If the experiment is so strictly defined and 
controlled that the relationship between the independent and dependent variables only holds 
in an extremely narrow context, the usefulness and value of the study is limited. 

The first type of generalizability that is often lacking in true experiments is external 
validity. External validity refers to whether the findings from a study would be valid in the 
external world in everyday natural settings. These concerns are especially there for lab-based 
experiments, which often lack mundane realism. Most lab spaces continue to be rather sterile 
environments that do not represent naturalistic settings. For social justice scholars who are 
seeking to challenge the status quo and reduce social inequities in the real world, it may be 
important that lab-based studies can later be translated into natural settings outside. There are 
numerous cultural and social factors that go into implementing any intervention or program 
that could be difficult to capture in a typical lab setting. However, lab-based experiments typi- 
cally provide high internal validity, which means that researchers can be quite confident 
that changes in the dependent variable are caused by the independent variable rather than 
extraneous factors. Such information can also be valuable to social justice researchers who 
would like to understand and measure specific factors that could bring about changes. 

Some ways to increase the generalizability of experimental research are to make laboratory 
settings as realistic and natural as possible. The laboratory tasks should resemble real-world 
tasks as closely as possible. Field experiments are a good alternative as they allow for observa- 
tion of behaviors in realistic and naturalistic settings. Also, whenever possible, researchers 
should include participants from the actual populations who would benefit from the study 
rather than whoever is most convenient to bring to the laboratory. Another way in which 
researchers could try to improve the external validity of a study is to manipulate more than 
just one independent variable to try to capture the nuances and complexities of communica- 
tion behaviors in the real world. For example, in the gaming study that we saw in the begin- 
ning of the chapter on portrayals of Black women, the researcher might want to manipulate 
both the gender and the race of the character to be able to capture how both these identities 
intersect in unique ways to inform us about racialized sexism. The downside, though, is that 
the experimenter would need additional time and sample size to work with more complex 
factorial experimental designs. In this case, if the research is interested in comparing White, 
Black, Latino/a, and Asian characters who are male, female, or nonbinary, then they would 
now have 3x4 conditions, making this a complex factorial experimental design. 

Another issue is with sample representativeness. Note that even if random assignment 
to conditions is used in an experiment, it is not necessary that participants are being recruited 
using probabilistic sampling. In fact, most experiments use convenience sampling, especially 
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of undergraduate college students, since they are an easily available group of participants for 
researchers at universities. However, if the variables that are being studied are experienced dif- 
ferently by college students compared to other groups, researchers also need to further study 
the phenomena with other non-student populations. This is not to say that researchers should 
not use convenience samples or college students as participants, but they should be aware of 
the limitations that such samples pose for the generalizability of findings. When it comes to 
implementing programs and interventions in a large-scale setting, it is especially important 
to involve participants from the communities where the program will be eventually imple- 
mented so that factors such as social networks, cultural differences, knowledge level, literacies, 
attitudes, and behaviors of the specific targeted group could be incorporated into the study. 


Summary 


Experiments help to establish causal relationships, test hypotheses, understand theoretical 
mechanisms, and test the effectiveness of interventions. The independent (or cause or predic- 
tor) variable is manipulated or introduced into the setting in an experiment. The dependent 
(or effect or outcome) variable(s) is always measured. Typically, experiments have at least two 
conditions: the experimental or treatment group and the control group. The control group 
is one that does not receive the treatment. Other variables in the experiment that are not 
the independent or dependent variables are called extraneous variables. They can be either 
control variables that the experimenter accounts for or confounding variables that are not 
accounted for in the study. In order to establish causation, there are three criteria: (1) there 
should be a significant association between the variables; that is, the variables should co-vary, 
(2) the independent variable should occur before the dependent variable, and (3) there should 
be no spurious relationship between the independent and dependent variable caused by an 
extraneous variable. A causal mechanism clarifies why the relationship between the independ- 
ent and dependent variable is observed. Mediating or intervening variables are explanatory 
variables that explain why the independent variable causes changes in the dependent variable. 
Moderating variables explain the context in which the independent variable causes changes 
in the dependent variable. 

True experiments are the best way to establish causality. True experiments must include at 
least two comparison groups, establish time order such that the independent variable preceded 
the dependent variable, and use random assignment to conditions. Random assignment to 
conditions means that a random process is used to determine which condition of the experi- 
ment the participant will take part in. When more than one variable is manipulated in an 
experiment, it is called a factorial experiment. Experiments could use posttest only design, 
pretest/posttest design, or Solomon four-group design. Quasi-experiments are similar to true 
experiments but lack random assignment to conditions and therefore, the comparison groups 
are nonequivalent. Pre-experiments do not include a control group, nor do they use random 
assignment to conditions. Field or natural experiments are those that take place in the real 
world outside the confines of a laboratory. Whereas lab experiments allow for careful control 
of extraneous variables, field experiments offer the advantage of realism in a natural setting. 

Participants for experiments, especially lab-based ones in university settings, have often 
been convenience samples of student populations. Sample size for experiments is determined 
by the number of experimental conditions, with factorial designs being complex and requir- 
ing more participants. Because experiments, by design, involve the manipulation of variables, 
informed consent and voluntary participation are crucial. In experiments, the independ- 
ent variable can be manipulated using stimulus materials, confederates, and role-play sce- 
narios. The measured dependent variables can be self-reported or observed. Traditionally, 
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experiments tend to be cross-sectional, but when conducted longitudinally, they are referred 
to as time series experimental designs. When random assignment is not possible, matched 
pairing could be considered for comparison groups. In controlling for extraneous variables, 
researchers should be cognizant of the Hawthorne effect, placebo effect, and experimenter 
effects. 

Debriefing refers to the final stage of the experiment in which researchers discuss the goals 
and purpose of the study as well as gather feedback, especially about discomfort or any stress 
experienced by participants. If deception is involved, it is crucial to acknowledge it during 
debriefing. Some of the threats to internal validity in experiments that researchers should try 
to avoid are history, maturation, testing, regression to the mean, selection threat, and attri- 
tion. To increase the generalizability of experimental findings, researchers should increase 
external validity by improving the realism of experiments and also pay attention to sample 
representativeness. 

Social justice scholars should be mindful of resources, context, timeframe, compensation, 
and effort as they determine the type of experiment that is best for their project. Creating 
culturally responsive and equitable distribution of benefits are two important considerations 
for social justice scholars using experimental designs. Overall, experimental methods are help- 
ful tools for social justice researchers, especially the use of interventions and programmatic 
initiatives, that help measure impact on relevant outcome variables. 


Discussion Questions and Activities 


1. What is the role of experiments in social science research? As social justice researchers, 
how can experiments be helpful to work toward the reduction of social inequalities and 
challenge the status quo? 

2. Lab-based experiments and field experiments are both invaluable approaches within the 
toolkit of a social justice researcher. When would you use each of these methods and 
why? What are the advantages and disadvantages of each type of experiment? 

3. Are random assignment and random sampling the same? How are they different? Which 
one contributes to internal validity and which one to external validity? 

4. What are some ethical considerations to keep in mind while designing and implementing 
experiments for social justice research? 

5. Lets imagine that you are a health communication scholar interested in studying the 
effectiveness of a campaign designed to promote healthy behavior (handwashing, flu vac- 
cination, safe sexual practices, exercising, anti-smoking, healthy eating, etc.). Identify the 
key independent variable(s), dependent variable(s), control variables, and other extrane- 
ous variables that are beyond the control of the experimenter. How would you design the 
experiment? How would you know if your findings are valid and reliable? 
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7 Examining Communication 
Content in Content Analysis 
Research 


In a sample of primetime and children’s TV content, girls or women comprised just 39% of 
characters with speaking roles (Smith, Choueiti, Prescott, & Pieper, 2012). African Ameri- 
cans were disproportionately depicted as more verbally aggressive within a sample of popular 
reality TV programs compared to those of other races (Glascock & Preston-Schreck, 2018). 
Within 20 hours of recorded game play from popular massively multiplayer online games, 
White males appeared much more frequently than women and characters of color (Waddell, 
Ivory, Conde, Long, & McDonnell, 2014). In a sample of television programs with lesbian, 
gay, or bisexual characters in leading roles, programs from streaming services like Netflix, 
Amazon Prime, or Hulu and premium cable providers like Showtime were more likely to 
feature sexual talk and behavior among LGB characters compared to broadcast or basic cable 
providers (Bond, Miller, & Stevens Aubrey, 2019). 

Claims and numerical estimations like these are made possible by the use of a quantitative 
research method known as content analysis. Content analysis is a social scientific method 
used to document the features of what appears in media content or in other forms of mean- 
ingful messages. Some content analyses focus, in particular, on entertainment-based, fictional 
stories told in the media, as illustrated in the opening paragraph of this chapter. Yet, content 
analysis can apply more broadly to other forms of media content, including posts or com- 
ments on social media, features of websites, or coverage of topics in the news. 

In fact, it can be applied to any form of communication content (i.e., any content with 
meaning) that can be gathered and examined for analysis, regardless of whether that content 
appears in the media. What do new employee training manuals from a set of organizations 
show about how to maintain a productive and supportive work environment? How do presi- 
dential speeches from the last ten years discuss the topic of income inequality? What do tran- 
scripts of doctor-patient interactions reveal about how health information is communicated 
to English language learners? The tools of content analysis can be applied to answer questions 
such as these. 

Both qualitative and quantitative scholars alike can be interested in studying the content 
of media or other messages. Applying some of the principles of research methodologies from 
Chapter 1, we can assume that when qualitative scholars study media content, they are likely 
to choose fewer texts or examples of content and explore that material in great depth and 
detail. When quantitative scholars study content, they are likely to choose more material to 
examine, and their observations are likely to be more summative in nature. But the most 
fundamental difference between qualitative and quantitative content analysis is that “quanti- 
tative content analysis has as its goal a numerically based summary of a chosen message set” 
(Neuendorf, 2017, p. 21). As we have certainly established by now, quantitative data are, by 
definition, numerical, and therefore observations of content must be represented by numbers 
in quantitative content analysis research. 
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Figure 7.1 Coders examine communication content to document featuress 


Source: Stephen Warren 


Let us look at some of the ways in which Journalism and Communication scholars have 
defined content analysis: 


e Klaus Krippendorff (2013, p. 24): “A research technique for making replicable and valid 
inferences from texts (or other meaningful matter) to the contexts of their use.” 

e Daniel Riffe, Stephen Lacy, Brendan R. Watson, and Frederick G. Fico (2019, p. 23): 
“The systematic and replicable examination of symbols of communication, which have 
been assigned numeric values according to valid measurement rules, and the analysis of 
relationships involving those values using statistical methods, to describe the communica- 
tion, draw inferences about its meaning, or infer from the communication to its context, 
both of production and consumption.” 

e Kimberly Neuendorf (2017, p. 17): “Content analysis is a summarizing, quantitative 
analysis of messages that follows the scientific method (including attention to objectiv- 
ity-intersubjectivity, a priori design, reliability, validity, generalizability, replicability, and 
hypothesis testing based on theory) and is not limited as to the types of variables that may 
be measured or the context in which the messages are created or presented.” 
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Each of these definitions reveals the logic of the scientific method at play in content analy- 
sis, as is seen in the definitions’ references to such topics as validity, measurement, assigning 
numeric values, replicability, and making inferences. The first two definitions include, as 
well, a reference to the purpose of content analysis: to understand what appears in texts as an 
important step in knowing why content takes shape and how audiences create, use, respond to 
and/or make meaning from that content. 


Using Content Analysis for Social Justice 


Content analysis research has the potential to lend insights into particular points in history by 
studying communication forms produced at a moment in time. Simply by describing media 
content or other sets of communicative content, content analysis research can implicitly reveal 
a great deal about the context in which that content or those messages were produced and 
disseminated. What was valued at the time? What was popular? Who was represented? Who 
was missing? 

Most hypotheses in content analysis research have to do with studying how one aspect 
of the content in question relates to another aspect. For example, you might have a 
hypothesis that predicts that Muslims will be more frequently depicted as threatening in the 
news compared to non-Muslims. There are two variables at play: (1) whether individuals in 
the news are Muslim or not and (2) the degree to which individuals are depicted as threaten- 
ing. In this example, you are predicting that there is a relationship between the religious iden- 
tity of individuals who appear in the news and the extent to which individuals are presented 
as threatening. The social justice implications of studies such as this are many, since they can 
be used to show biases in media or other communication content. 

Content analysis research might seek to understand how media content compares to 
observations drawn from the wider world. Rather than interpreting media content as 
implicitly related to social or cultural or political factors, the researcher can explicitly study the 
relationship between those factors and media content. In Scharrer, Kim, Lin, and Liu (2006), 
for example, the performance of domestic duties among real-world heterosexual couples in 
the United States was compared with the depictions of household chores in commercials. 
Real-world statistics showed that within heterosexual couples, men were doing more to help 
out around the house (although women were still doing the most). Within the commercials 
examined through content analysis, the authors (including the first author of this book) found 
nearly twice as many female characters engaged in household chores than male characters. 
Thus, depictions in commercials were out of step with modern realities of U.S. households 
in gender stereotypical ways. Using content analysis in this manner can shed light on social 
justice related topics in that these observations can reveal distortions between communication 
content and a wider reality. 

Finally, many researchers use the results of content analysis to speculate about the out- 
comes that might stem from the depictions that have been established through the study 
of content. Do stereotypes of social groups found in the media contribute to prejudice among 
media audiences? An important first step in exploring this issue is figuring out how preva- 
lent those stereotypes are in media content through content analysis. One must be cautious, 
however, not to try to answer a question about effects of media content on audiences through 
content analysis research. A content analysis only tells us about the messages themselves, not 
how individuals might respond to or be influenced by those messages. 

If the researcher is not satisfied with speculating about what effects might be spurred from 
particular content features, that researcher may conduct a follow-up study that explores media 
effects. Perhaps the content analysis findings can identify a text or message to use as a stimulus 
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in an experiment designed to measure impact of media exposure. James A. Anderson (2012) 
suggests: 


Content analysis can play an important (and often neglected) part in audience effects 
studies. . . . Most effects studies are rather cavalier in the selection of content with the 
researcher selecting the content with an ‘of course it’s true’ approach that this content is 
violent, sexy, gender based or whatever the issue. It would be much better if we adopted 
a more sophisticated analysis of the content and could point to the properties by which 
the genre claim is made. 


(p. 279) 


Overall, the use of content analysis can be an important step in challenging the status quo. 
Yet, simply revealing the state of affairs in communication content does not actually work 
toward changing it. Content analysis on its own cannot achieve goals associated with social 
justice unless researchers and other individuals decide what to do with the information that 
content analysis reveals. On its own, content analysis is likely to spur the sort of “third-person 
scholarship” that is about but does not actively intervene in social injustices that Frey and 
Carragee (2007) introduced. In order to move content analysis research from third-person to 
first-person social justice research (Frey & Carragee, 2007), the researcher will likely have to 
ensure that a broader public beyond just fellow researchers can learn about and act upon con- 
tent analysis results. We will take up some of these strategies in the final chapters of this book. 

From this discussion, you are likely to be thinking that there are both strengths and weak- 
nesses to content analysis in general as well as for its use as a means toward social justice efforts. 
Indeed, one of its chief strengths is that it is not obtrusive. Since it does not involve studying 
human beings but rather is about studying media content or other messages that already exist, 
the researcher’s presence does not affect the observations made. Rather, the content existed 
before the researcher chose to analyze it, and therefore it is not subject to the sort of reactivity 
that we see in other forms of research. Content analysis research also tends to be low in cost, 
and it relies on fewer people to be carried out compared to surveys or experiments. As we will 
see, the researcher just needs at least one other person with whom to compare observations 
of content in order to conduct a quantitative content analysis. 

A shining example of the strength of content analysis research for social justice purposes, 
specifically, can be seen in the work of Stacy L. Smith, director and founder of the Annenberg 
Inclusion Initiative at the Annenberg School for Communication and Journalism at the Uni- 
versity of Southern California. Dr. Smith and her team conduct content analyses that docu- 
ment diversity, or lack thereof, in music, television, film, and other forms of entertainment. 
They also track the diversity of the people who create that entertainment media content, 
drawing a link between who is represented in the decision-making circles of entertainment 
media creation and who is represented in the resulting content. Furthermore, Dr. Smith 
and colleagues do not simply publish these data for academics to read. Rather, they engage 
directly in advocacy and action with media creators and entertainment industry decision 
makers to work toward positive change. You can read more about this initiative—built on 
the foundation of quantitative content analysis research—here, https://annenberg.usc.edu/ 
research/aii#aii-team. And you can read more about the role of quantitative research in advo- 
cacy and activism in the last chapter of this book. 

In terms of weaknesses, content analysis research is limited to forms of messages that are 
recorded and can be accessed or acquired by the researcher. There is also a limit to the sorts 
of questions you can ask and answer using content analysis methodology. As we have seen, 
although you can speculate about how people may interpret or be affected by messages, or 
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even why messages take the shape that they do, content analysis on its own does not provide 
data on those questions. Finally, with content analysis extending into social media posts or 
other user-generated content, some have debated the ethics of studying content produced 
by people without their knowledge or permission to do so, even if that content was posted 
publicly. 


Coding and Coders 


At the heart of content analysis research is the process of coding, defined as applying defini- 
tions of concepts to a set of texts or messages in order to document the features of those texts 
or messages. Put differently, the process involves “the systematic assignment of communica- 
tion content to categories according to rules” (Riffe, Lacy, & Fico, 2005, p. 3). To study the 
features of media content or other messages, therefore, it is necessary to know what you are 
looking for and how you will know it when you see it. The content analysis researcher comes 
up with rules regarding how to classify messages into meaningful patterns. 

Coders are the individuals who actually do the coding, making judgments about how 
the content examined matches up with the established rules regarding how to classify it. The 
concepts that will be examined in the content become variables in the content analysis study. 
Each variable has its own carefully determined set of rules regarding how to identify and 
classify content pertaining to that variable. The researcher comes up with the rules that will 
govern how the coders make coding decisions. The researcher also trains the coders in how 
to apply the established rules to the content in the sample to make valid and reliable coding 
judgments. 

Historically, only human coders performed content analysis coding. People—sometimes 
paid by the researcher, sometimes volunteers, sometimes the researcher or members of the 
research team themselves—were trained to apply the definitions of concepts to content and 
thereby carry out the coding. Indeed, that approach is still quite common to this day. Yet, 
more recently, it has become possible to use computer algorithms or programming to detect 
particular features of content, and therefore today there is automated coding in studies of 
communication content, as well, sidestepping the need for human coders. We will focus most 
of our attention in this chapter on human coding because automated coding is still a rather 
specialized skill, but we will intersperse some discussion of automated coding, as well. 

Most variables studied in content analysis research will take one of a few possible forms. 
Some variables are measured in a manner that simply determines whether the variable is 
present or absent. Imagine you are studying 300 ads for video games that appeared in 
print magazines. In each ad, are characters shown displaying aggression? Coders looking at 
that variable may simply code 0 = no or 1 = yes for each character. That way, the coder is 
determining whether aggression was present or absent for each character. Still other variables 
are about the degree to which a concept is present or absent. Within a sample of news 
content, for example, the researcher might want to know how much graphic detail of inju- 
ries or death is included in news footage of drone warfare. Researchers could study this on a 
scale of “1: not graphic at all” to “5: very graphic,” thereby capturing not just whether such 
detail is present but the degree to which it is. Counts (or frequencies) are produced when 
the number of times a variable is present in the content is tallied. For example, the number 
of times each presidential speech mentions income inequality is a count variable that can be 
examined in a content analysis. Such variables are likely to vary, then, from 0 (no references 
to income equality at all, in our example) to whatever the maximum number may be within 
the sample. 
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Each of these types of variables, whether they record presence or absence of something, degree 
to which something is present or absent, or the frequency with which something is present, will 
easily translate into a number. Yet even aspects of media content or other messages that do not 
readily translate into a number can, indeed, be quantified. You can categorize content into more 
descriptive categories, as long as you assign numbers to those categories. Imagine that you want 
to use transcripts to study differences between doctor-patient communication in contexts in 
which patients do or do not have English as their first language, for instance. To do this, you will 
have to classify each patient into the category of “English as first language” or “English language 
learner.’ To quantify this variable, you would assign numerical values to each option, such as 
1 = English as first language, 2 = English language learner. (Of course, you might need more than 
those two categories. What if it is not clear whether the patient is an English language learner, for 
instance? You might need a category such as 3 = unknown/uncertain.) 


Steps Involved in Content Analysis 


Content analysis follows a standard set of procedures designed to work toward the twin goals 
of reliability and validity. A key starting place is with a rationale for why you are studying 
particular types of content for particular observations and a comprehensive literature review 
so that you can see how former studies inform your study. From this stage, hypotheses and/ 
or research questions should emerge, which will guide testing of whether theorized obser- 
vations or relationships between variables occur in the content you are studying. 

The next step is conceptualization of variables, following the principles and practices that 
we outlined in Chapter 3, providing dictionary-type definitions of variables to be examined 
in the study. As Neuendorf (2017) suggests in her book on content analysis, when begin- 
ning to conceptualize the variables to be examined in a content analysis study, you may want 
to explore some of the content you are proposing to study, to see if you have missed any 
important aspects of that content that you want to include in your study. If you are studying 
presidential campaign ads, you would be wise to take a preliminary look at a bunch of those 
ads to help you figure out what it is about the ads you find interesting or important in the 
conceptualization stage of your study. 

The content analysis researcher then moves from conceptualization to operationalization, 
which means that the conceptual definitions now need to be translated into specific ways to 
measure the variables at play in the study. At this stage, it is important to remember to apply 
what you have learned about operationalizing variables in a manner that includes coming 
up with values that are exhaustive and mutually exclusive, and to think about how complex 
concepts might have multiple dimensions and/or indicators, again as we learned in Chapter 3. 

An important aspect of operationalization is determining the unit of analysis in the 
study. The unit of analysis is “the element on which data are analyzed and for which findings 
are reported” (Neuendorf, 2017, p. 20). For a study of television commercials, for instance, 
each commercial will likely constitute “the element on which data are analyzed.” For a study 
of social media posts, each post could be the unit of analysis. Even though this seems quite 
straightforward, determining one or more units of analysis can actually be quite complex in 
practice, and so we will return to this stage in the process a bit later in this chapter. 

Operationalized variables should then each be fully described and explained in a coding 
scheme or codebook. The codebook functions as a training manual for coders and contains 
detailed instructions for how to treat each variable in the study. It contains everything that 
coders will need to know to apply the operational definitions to the content in question in 
order to assign the numerical values associated with each variable. 
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Some researchers also create a separate document known as a code sheet at this stage in 
the process. Whereas the codebook explains how to do the coding, the code sheet is the place 
where each coding decision is recorded. Other researchers opt to forgo this step and simply 
record their observations right into an Excel spreadsheet, SPSS file (with SPSS standing for 
Statistical Package for Social Sciences, a popular software for statistical analysis in Commu- 
nication), or other data management programs. We have included an example of a content 
analysis codebook and associated code sheet here. 


Research in Action 


This is an excerpt from the coding scheme used for Scharrer, E. (2012). More than just 
the facts? Portrayals of masculinity in police and detective programs over time. Howard 
Journal of Communications, 23, 88-109. doi:10.1080/10646175.2012.641882 


MASCULINITY AMONG MAIN CHARACTERS IN POLICE/ 
DETECTIVE DRAMAS OVER TIME 


Character is the unit of analysis. You need one code sheet per character. We are only coding 
major characters, defined as characters who are in the credits, are in most episodes of the 
program and/or in most scenes in the episode, and/or are featured in a single episode as the 
main suspect or criminal. We are only coding adults, which means those who appear to be 
18 or older. Finally, we are only coding men and/or male-identified characters. 


I. Coding characters: First, list the name of the character you are coding. If you do 
not know the name, use a brief description to identify the character. 


1. Please use the following categories to code the apparent race or ethnicity of 
the character, using visual cues, any identifying dialogue, or any other indica- 
tor from the character or other characters. In the case of Latino characters, this 
might include their name. 


1 = White/European American 

2 = Black/African American 

3 = Latino 

4 = Asian/Asian American 

5 = Native American 

6 = biracial or multiracial 

7 = unknown/unable to determine 


2. Role in criminal justice process: Based on the narrative and any other 
indicator from the episode, what is this character’s role in the crime/criminal 
justice process? 


1 = police or detective 
2 = other law enforcement or legal figure (district attorney, defense attorney, etc.) 
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3 = witness 


4 = suspect 
5 = criminal 
6 = other? 


3. The character’s display or discussion of emotions: We are interested in 
whether the character expresses emotions either visually (in their actions, on 
their faces, in what they do) or verbally (in what they say). 

3a. Shows emotions visually: Please code whether the character displays any of 
these emotions by having the emotion registered on his face or apparent in his 
actions (e.g., he slumps to the ground or cries in sadness, widens eyes or quick- 
ens breath in fear, smiles or laughs to show happiness, slams the steering wheel 
in the car in anger). If there is lack of sincerity or if there is sarcasm, please do 
not count it. Only count sincere emotions displayed. 


Happiness? Y or N 
Sadness/grief? Y or N 
Anger? Y or N 

Fear? Y or N 


3b. Shows emotions verbally: Please code whether a character talks about an emo- 
tion or verbally expresses a state of mind/being. This can be indicated both by 
what he says (e.g., “You are really pissing me off’) and the manner in which he 
says it (e.g., yelling in the case of anger, or with a cracking or wavering voice 
in the case of sadness). Again, please only count sincere emotions spoken. 


Happiness? Y or N 

Sadness/grief? Y or N 

Anger? Y or N 

Fear? Y or N 

Sympathy or empathy for someone: Y or N 
Guilt/remorse: Y or N 


3c. Overall, how emotionally responsive would you say this character is, judging 
from his depiction across the entirety of the episode? A low score would indi- 
cate reticence or lack of expression of emotions, whereas a high score would 
indicate either/or a wide range or a frequent display of emotions. 


1 2 3 4 5 
Stoic Emotional 


Sampling is the next stage, and it follows the principles and practices we discussed in 
Chapter 4. In content analysis research, as in other quantitative methods, if you can employ a 
random sampling method, it is well worth doing so, since that will allow you to estimate with 
precision the likelihood of obtaining a particular result in the sample without it also being 
present in the population. Yet, when that is not possible, any of the nonprobability methods 
described in Chapter 4 are available to guide you in the collection of a sample that is system- 
atic and relatively unbiased in nature. We will explore the unique opportunities and challenges 
of sampling in content analysis research a bit later in this chapter. 
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Once the codebook has been carefully developed and the sampling strategy is set, it is time 
to train coders. Quantitative content analysis must employ at least two coders. The underly- 
ing principle is that the observations made in the content analysis process are not merely one 
person’s opinion or unique interpretation but rather should be able to be detected consistently 
by multiple people (at least two) using the same coding procedures. Importantly, the coders 
should be working independently from one another, so that one person’s judgments regarding 
how to apply the coding scheme do not influence the other’s judgments. 

Intercoder reliability (ICR) is the degree to which multiple coders are applying the 
coding scheme consistently and agreeing with one another regarding how to code variables. 
In order to test for ICR, two or more coders code the same material so that a mathematical 
test of their agreement on that shared content can be conducted. Again, you will read more 
on this topic later. 

At this stage in the content analysis process, the pilot testing of intercoder reliability is 
typically conducted, not on the actual sample itself but on a pilot test sample that is similar in 
nature to the actual sample. If you have gathered a sample for a content analysis that consists 
of 250 presidential campaign ads, for example, you should set that sample aside and not have 
coders code it until they have pilot tested the coding scheme using presidential campaign 
ads not in the sample. Riffe and colleagues explain why this is the case, “If content is coded 
several times, prior decisions contaminate subsequent ones. Furthermore, the effect of multi- 
ple codings is to inflate the ultimate reliability estimate, thus giving a false confidence in the 
study’s overall reliability” (Riffe et al., 2005, p. 142). The pilot test(s) of ICR, then, should be 
conducted independently from the final test of ICR. 

Intercoder reliability testing during the pilot testing stage may result in the realization that 
the coding scheme needs to be revised for greater clarity and accuracy in coding. You may 
find that coders do not agree with whether a variable is present or absent, for instance, or are 
otherwise having trouble determining how to code a variable. The coding scheme will then 
need to be clarified and intercoder reliability retested to ensure that there is broad consensus 
regarding how to code each variable. Once the coders are in syne and the codebook is work- 
ing well in producing reliable observations across coders in the pilot stage, the official coding 
of the actual sample can take place. 

The official test for ICR is the next step. The rule of thumb is that at least 10% of the 
official sample for the study should be coded by two or more coders to test for intercoder 
reliability. More than 10% overlap between or among coders is even better. Once again, the 
coders should be working independently, so as not to influence each other. As long as the 
results of that test pass muster, the coders can independently code the rest of the content with- 
out having to overlap in what they are coding, because they have shown that they are doing 
so in a systematic and reliable manner. We will explain how to determine whether intercoder 
reliability has been established in more detail later in this chapter. 

Finally, the content analysis data are analyzed and reported. This usually takes the 
form of statistical analysis and should test hypotheses and explore research questions, thereby 
providing a mathematical answer to whether hypotheses are supported or refuted and how 
research questions can be answered. We will go into great detail about data analysis in subse- 
quent chapters of this book. 


Making Sound Observations 


For any quantitative research method, validity and reliability are main concerns. This is cer- 
tainly the case in content analysis. Ways to code variables are valid if they are accurate in 
reflecting the meaning of the concepts the variables are meant to capture. Ways to code 
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variables are reliable when applying the same coding definitions repeatedly produces very 
similar results every time. In content analysis research, reliability is largely determined through 
ICR. 

Sometimes a content analysis researcher will have to balance richness and detail in coding 
with the likelihood or ease with which content analysis coders will agree. Simple, straightfor- 
ward variables will clearly be easier to agree on, and therefore coding those variables is likely 
to be reliable. Yet, confining a content analysis to simple, straightforward variables is also likely 
to limit what can be revealed about the content. Complex variables can be challenging—but 
not impossible!—to develop into a reliable coding process. 

Imagine you are doing a study that involves determining whether each music video within 
a sample of music videos contains violence. For each video, you could ask coders “Is this 
violent?,” and they could potentially have two values to use for their observation: Yes or No. 
Or you could ask coders, “How violent is this?,’ and provide three values for that variable: 
1 = not violent at all, 2 = somewhat violent, 3 = violent. In still another possibility, you could 
ask, “How violent is this?,’ and have five possible values for that variable: 1 = not violent at 
all, 2 = a bit violent, 3 = moderately violent, 4 = quite violent, 5 = very violent. Clearly, it 
will be easier for multiple coders to agree on the first item, followed by the second item, and 
then the third. The more fine-grained the coding judgment, the more difficult it usually is to 
find reliable results. Yet, the more fine-grained measures have the potential to tell you more 
about the content than the ones that use broader categories. Of course, they only do that if 
they can be reliably coded. 

Relatedly, some coding calls for observing manifest content whereas other coding calls 
for observing latent content. Manifest content involves variables that are apparent on the 
surface of the content in question. They are overt, there for the eye to see or the ear to hear. 
For example, you could code whether or not presidential candidates are shown wearing busi- 
ness suits in campaign ads. All coders with a shared definition of what a business suit is could 
look at the content as it exists, on the surface, and determine whether a candidate in an ad is 
shown wearing one. Latent content is under the surface. These variables require inference 
to code, demanding that the coders make an inferential leap from something observable (i.e., 
more manifest) to a related judgment. For example, rather than having coders note whether 
a presidential candidate is wearing a business suit, you might code whether the clothing they 
wear in the campaign ad is gender stereotypical. Coding this variable involves making an 
inference from what is on the surface (what the presidential candidate is wearing in the ad) to 
a more complex judgment (whether that clothing would be considered gender stereotypical). 

Latent content, as you might guess, is more difficult for multiple coders to agree upon, 
given that it is more subject to interpretation. And yet, if latent content variables do, indeed, 
stem logically from observable content and are well articulated in the coding scheme, they 
can be reliably coded within a reasonable range of agreement by multiple coders. And they 
often have a great deal to contribute to the study. For our presidential campaign ad example, 
for instance, we would need a careful definition of what constitutes “gender stereotypical” 
clothing, and if we could arrive at one, using that variable would reveal a great deal about the 
content. 

Consider this example of the ways in which Teresa Lynch, Jessica E. Tompkins, Irene I. van 
Driel, and Niki Fritz (2016) studied sexualization of female video game characters by exam- 
ining the appearance of the main playable female character within 571 video games. Their 
operationalization included examining a visual emphasis on particular body parts and the 
ways in which characters moved. For each of the five variables listed, Lynch and colleagues 
studied whether that depiction was present or not present, assigning a score of 0 for not pre- 
sent or 1 for present. Then they combined these items into a single measure of sexualization 
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by summing up each of the five individual observations. The parenthetical information that 
includes the Greek letter alpha (a) shows the score for each variable on Krippendorf’s alpha, 
which—as we will explain later in this chapter—is a common measure used to report reli- 
ability among multiple coders. 


We considered chests sexualized if we observed one or more of the following: breasts dis- 
proportionate to the body size (a =.81), bare skin between the armpits and bottoms of the 
breasts (a =.70), or accentuation by garments or artistic styling (e.g., shading; a =.73). We 
observed sexualized buttocks if skin was exposed from the top of the hips to the bottom of 
the buttocks (a =.71) or if adornments or art styling accentuated the buttocks (a =.74). We 
considered waists sexualized if the character’s midriff area in the front or back of the body 
was bare (a =.80) or if the waist to hip shape was exaggerated (a =.78). We considered the 
leg region sexualized if the skin on the legs was exposed from the hip area to the top of the 
character’s knees (& =.95). Finally, we determined the presence of sexualized movement 
(a =.75) if the character’s movements included unnecessary undulation or jiggling that 
drew attention to their body in a sexual manner. 

(p. 572) 


In this example, then, we see how an abstract concept like sexualization can be broken down 
into multiple, specific indicators that are observable within the content. Lynch and colleagues 
may not have anticipated every possible way of studying sexualization. Yet the manner in which 
they decided to code the variable does appear to be valid—given that it maps on to broad under- 
standings of what sexualization can look like in video games—and is likely to be reliable—given 
the specificity and overt nature of and the “present” or “absent” way of coding each indicator. 
The researchers have taken an abstract concept and made sure it could be coded based on mani- 
fest content, thereby minimizing the inferences and other judgments the coders had to make. 


Sampling Considerations in Content Analysis 


As you read in Chapter 4, constructing a sample on which to base observations is an important 
step in quantitative research. In content analysis, researchers must determine where and how 
to find the TV programming, advertisements, websites, video games, social media posts, movie 
trailers, magazines, news articles, and other content that will be examined in the study. As we 
also learned in Chapter 4, they must also decide how much of that content is necessary in order 
to be fairly confident that observations based on any given sample can be generalized to a wider 
body. The same probability and nonprobability sampling techniques explained in Chapter 4 are 
available for content analysis researchers to employ, and the same overall goals of purposiveness 
(meaning that the content is chosen for a reason) and/or representativeness in sampling apply. 

Sampling in content analysis research is often constrained by access. As you will recall, in 
order to use random sampling techniques, a comprehensive list of population units must be 
generated. Sometimes in content analysis research, you may be able to come up with a com- 
plete list of content to sample, but you just cannot get your hands on some of that content 
in order to code it. Hugh Klein and Kenneth F Shiffman (2006), for instance, went to great 
pains to access all cartoons using cel animation produced by the major animation studios 
between 1930 and the mid-1990s in their study of racial diversity among characters and overt 
depictions of racism. They describe their process: 


Once the “universe” of cartoons had been identified, actual copies of the specific car- 
toons selected for viewing and coding as part of the random sampling approach had 
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to be located. This was done in a wide variety of ways: by contacting animation fans 
and collectors and having them make copies of some of their cartoons for us, visiting 
film archives and repositories and viewing cartoons in their libraries/holdings on site, 
obtaining copies of the needed cartoons directly from the animation studios, purchasing 
sample-selected items from retail outlets and private sellers who advertised them in trade 
publications, renting videocassettes from retail outlets like Blockbuster Video, and vide- 
otaping from programs broadcast on television. 

(p. 168) 


This is clearly a great deal of effort, but, in this case, it paid off in terms of facilitating a 
comprehensive look at the phenomenon in question. 

Electronic databases are excellent resources that allow researchers to access particular 
texts or messages. If your library has access to the LexisNexis electronic database, for example, 
this is a treasure trove for the study of news content or legal documents. Within the database, 
you can search under keywords, within particular ranges of dates, and within particular pub- 
lications. When the first author of this book studied the ways in which Hillary Clinton was 
depicted in the news when she was first lady, LexisNexis was used to identify news stories 
featuring Clinton, and then a systematic sample was drawn from the resulting list. “To con- 
struct the sample, the search term ‘Hillary (Rodham) Clinton’ was entered for the time period 
of 1 October 1999 to 6 February 2000 in the LexisNexis database. From the list of sources 
displayed, every fourth story was selected to reach the ultimate sample size of 342 stories on 
Clinton” (Scharrer, 2002, pp. 396-397). 

Similarly, content analysts can use archives to access content for sampling and analysis. 
In yet another example from the first author of this book, for instance, police and detective 
programs that aired on TV from the 1950s through the 1990s were accessed at the Museum 
of Television and Radio in New York City for a study of whether the male main characters 
exhibited characteristics associated with “hypermasculinity” (Scharrer, 2012). The archive at 
the Museum of Television and Radio provides access to a number of programs from across 
the decades that otherwise would be very difficult for a researcher to find. The Research in 
Action section of this chapter shows an excerpt from the codebook for that same study. 

Although databases and archives present rich opportunities for the content analyst, those 
sources are not without limitations. They may not house every title or all content of interest 
to the researcher, for instance, thereby limiting the generalizability of what can be found. If 
you use the holdings in the database or archive as the sampling frame from which to make 
sample selections, therefore, you have to be mindful that the sampling frame is not the same 
as the population. Researchers using these tools must be careful not to generalize further 
than the holdings allow. For example, in Scharrer’s (2012) study of police and detective 
programs, the goal was to examine every top-rated police and detective program from the 
beginning of television in the United States to the end of the 1990s. Yet, some titles that 
were listed as top-rated programs were not held in the archives, were not accessible through 
any other known means, and therefore did not make it into the sample. Yet another limita- 
tion is that sometimes the use of archives or databases incurs a cost that the researcher may 
not be able to sustain. 

Using a composite sample (also known as a constructed sample) is a variation on ran- 
dom sampling techniques that is a popular and rigorous approach used in many studies of 
media content. This approach uses the circulation or programming patterns of media content 
as the main logic to guide sampling. For television programming, which is organized around 
time slots, a composite sample can be randomly selected from a list of time slots across days 
of the week and channels. Mastro and Figueroa-Caballero (2018), for example, studied a 
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composite week of scheduled television programming in their analysis of the body size and 
shape of characters that appeared in primetime television programming. They report: 


Using a simple random sample without replacement, a 1-week composite of current 
prime time television programming (8-11 p.m. PST) across the nine broadcast and cable 
networks with the highest English-language viewership in the United States (ABC, 
AMC, CBS, CW, FOX, NBC, USA, TBS, & TNT) was constructed over a 10-week 
sampling period from September to December 2013. Only scripted and reality television 
programs were included. Special events (e.g., American Music Awards), sports, and news 
were omitted. In order to identify the units in the population (i.e., programs on prime 
time television), a list of all programs appearing on each of the selected networks dur- 
ing the regular 2013—2014 television season was compiled. One episode of each of the 
scripted entertainment programs within the population was randomly sampled, using a 
random numbers table, during the 10-week timeframe. This resulted in a final sample of 
89 programs and 1,254 distinct characters. 


In this example, then, imagine a list of all ABC programs that air between 8 pm and 11 
pm. Then do the same for AMC programs, followed by CBS programs, etc. until you have 
exhausted all nine of the channels Mastro and Figueroa-Caballero (2018) used in their study. 
The composite week involves randomly selecting across each of the time slots and each of the 
channels to come up with one week’s worth of primetime programming created from each 
of those randomly selected titles. 

Finally, content analysis research can be used to test for stability or change over time by 
using a longitudinal sample. Both the Scharrer (2012) study of hypermasculinity in police 
and detective programs and the Klein and Shiffman (2006) analysis of race and racism in ani- 
mated cartoons discussed as examples in this section involved sampling across time periods to 
make observations about whether depictions had changed. Once again, access proves essential 
here since older content can often be more difficult to find. 

We have provided an example (Figure 7.3) of observations of how the regular cast of char- 
acters on broadcast television has changed over time based on content analysis research issued 
by the organization, GLAAD. You can find reports on the methodology used by GLAAD 
and other excellent resources for media representation across the gender and sexual identity 
spectra at the website, www.glaad.org/tags/where-we-are-tv. The organization issues annual 
“Where We Are on TV” analyses as well as other reports of media representation using both 
quantitative and qualitative approaches. 


Determining Units 


An important consideration in content analysis research is how to divide up the content that 
will be coded into manageable units. Within the communication content to be studied, which 
portions or aspects of the content, exactly, will be examined to apply the coding scheme and 
make content analysis observations? Recall that “The unit of analysis is the element on 
which data are analyzed and for which findings are reported” (Neuendorf, 2017, p. 20). 
Sometimes determining the unit of analysis is rather straightforward. For a study of music 
videos, for instance, it might make sense for each video to be the unit of analysis, meaning 
that you will apply the coding scheme across the content of the videos in their entirety, one 
video at a time. Other times, determining the unit of analysis can be more complex. For 
example, you may have to determine how fine-grained your unit of analysis will be. Within 
a study of newspaper content, will the entire newspaper be analyzed and therefore constitute 
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the unit of analysis? Or will articles within the newspaper be the unit of analysis? When cod- 
ing television or film content, will the unit of analysis be the program or film, scenes within 
the program or film, or characters within the scenes? In any of these cases, you choose the 
unit of analysis that makes the most sense for observing the variables you are coding. 

Some content analyses have more than one unit of analysis. For example, Dana E. Mastro 
and Susannah R. Stern (2003) employed two units of analysis in their study of racial represen- 
tation in commercials aired on U.S. television during prime time. The commercial itself was 
the unit of analysis for many of the observations they made to code variables. Yet, they also 
made observations at the level of the character within the commercial, coding the first three 
speaking characters who appeared. Coding at both levels of analysis enabled Mastro and Stern 
to study race as a character-level variable and product type and setting as commercial-level 
variables (among many other variables). This allowed the researchers to find, for instance, that 
Asian characters appeared most often in ads for technology and were shown most often at 
work, thereby revealing the presence of racial stereotyping. 

Some content analysis research designs will also call for a unitizing step, which involves 
making a judgment call about whether potential units within a “continuous stream of infor- 
mation” (Neuendorf, 2017, p. 72) should be included in the sample or not. If you are con- 
ducting a study of whether cyberbullying posts in social media show that individuals have 
been targeted based on some aspect of their identity, for instance, the first step might be to 
figure out whether a post contains cyberbullying at all. From a stream of social media posts, 
then, relevant units must be selected—in this case, those that contain cyberbullying—based 
on a judgment of whether they meet the definition to warrant inclusion in the sample. 

Sometimes, the researcher may unitize content for the coders, handing over the coding 
in units the researcher has already determined to be relevant. Yet, a more rigorous approach 
is for coders themselves to determine whether a unit within a continuous stream meets the 
definition set forth by the researcher to be relevant for inclusion in the sample. In fact, it is 
considered best practice to report reliability scores (explained in the following section) for the 
unitizing step in the content analysis. 

The first author of this book, for example, co-authored a study on the ways that father 
characters have been portrayed on U.S. sitcoms over time (Scharrer et al., 2020). Scenes in 
which sitcom fathers were involved in disparagement humor, defined as either making fun 
of someone else or being made fun of oneself, were the unit of analysis. From the continu- 
ous stream of sitcom content, coders first identified the scenes that involved the father in 
disparagement humor and therefore were included in the sample. Once that set of content 
was identified, the coders then applied to the content a host of variables pertaining to how 
the sitcom father was portrayed. 


Intercoder Reliability 


As we have mentioned, establishing intercoder reliability (ICR) is an essential part of conduct- 
ing a successful quantitative content analysis. If the ways of classifying or otherwise coding 
content that have been developed by the researcher are sufficiently clear, complete, precise, 
and detailed, then the coders’ ability to apply the coding scheme should, for the most part, 
supersede human biases or individual ways of interpreting the content. Thus, the overall 
concern for reliability in research—the likelihood that findings are stable and therefore repli- 
cable—is largely addressed by intercoder reliability in content analysis. In other words, “The 
problem of assessing reliability comes down ultimately to testing coder agreement to verify 
the assumption that content coding is determined by the concept definitions” (Riffe et al., 
2005, p. 123) rather than resulting from one person’s opinion or interpretation. 
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This, we would argue, is a key facet of the power of numbers. If you are advocating for 
more diverse representation in Hollywood films, in television programs, in video games, etc., 
your argument is likely to be more convincing if you can show that multiple coders agree that 
the present depictions are limited in terms of number or quality. The logic of content analysis 
research is that any two or more coders trained to apply the coding scheme to the content 
will more or less agree about how to classify or otherwise describe that content. Coders can 
be students, friends, relatives, partners, colleagues, or really just about anyone who is able to 
understand how to apply the coding scheme to the content and motivated to be conscien- 
tious about doing so. Content analysis data can be persuasive, then, in that they indicate that 
there is, indeed, something about the content in question that is able to be ascertained by just 
about anybody and therefore cannot be written off as a biased or idiosyncratic interpretation. 

Content analysis experts suggest that at least 10% of the overall sample should be coded by 
multiple coders and subject to a formal test of ICR. Recall, as well, that the coders should be 
working independently from one another so as not to sway the others’ coding decisions. If the 
result of that test of ICR comes back at an acceptable level, multiple coders can then divvy up 
the rest of the sample, because they have shown that they are using reliable means of applying 
the coding scheme to the content. 


Scholar Spotlight: Dr. Travis Dixon 


Travis Dixon (PhD, University of California Santa Barbara) is Professor and Communi- 
cation Alumni Professorial Scholar in the Department of Communication at the Uni- 
versity of Illinois. Dr. Dixon studies both media content and media effects, particularly 
regarding the presence of stereotypes in media and the influence of stereotypes on media 
audiences. Across multiple content analyses, Dr. Dixon has studied race and crime in the 
news, including through analyses that compare news content with social statistics such as 
crime reports. He has also studied the presence of stereotypical content such as violence 
and misogyny in addition to more positive, prosocial themes in rap lyrics. 


Q: What motivates you to do content analysis research in general, and to study the issues and 
topics at stake in your content analysis work, in particular? 

Dr. Dixon: I think that it has to do with my casual observations and experiences grow- 
ing up in South Los Angeles. LA is a diverse and yet quite segregated city where we 
still tend to stereotype one another. I speculated, even as a young student, that we 
received biased messages from the media that reinforced these stereotypes. Content 
analysis offered me a methodology to explore this as a more mature scholar. 

Q: What challenges arise in your research pertaining to key steps in content analysis like sam- 
pling, intercoder reliability, or operationalization of concepts? 

Dr. Dixon: Content analysis is the most labor intensive of all the quantitative social 
scientific methodologies. Rigorous content analyses rely on multiple coders who 
have to reach high levels of intercoder reliability. Coder training and achieving 
sufficiently high reliability are the most difficult challenges associated with content 
analysis. 

Q: What are your experiences and/or your hopes regarding how your research can be used to 
challenge social injustices and help make the case for social change? 
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Figure 7.4 Scholar Spotlight: Dr. Travis Dixon 


Dr. Dixon: My hope is that think tanks and social justice advocates will use the data 
generated by my research to make an argument for positive social change. My 
goal is to write in an accessible way for both social scientists and advocates to help 
facilitate this process. 

Q: Do you think quantitative methods in general and content analysis in particular can be tools 
to work toward social justice? Why or why not? 

Dr. Dixon: I definitely believe that quantitative methods, including content analysis, can be 
used in the work toward social justice. This work can be used to educate policy makers 
and the public at large. In a democracy access to good information remains extremely 
important for advancing debate and to have lawmakers propose useful legislation. 


So, how should the researcher select the 10% or more of the sample that will be coded by 
multiple coders and tested for ICR? Some experts suggest randomly selecting that subsec- 
tion of the sample. Indeed, based on the principles of sampling that we have described in this 
book, that constitutes a best practice. If taking a random sample is somehow difficult to do, 
however, the guiding principle is that the subsample tested for ICR should be sufficiently var- 
ied in nature to ensure that the entire coding scheme can be applied. In other words, it should 
contain content that corresponds to all of the variables in the study, so that each variable can 
be fully considered by coders and tested for intercoder agreement. 

There are a number of mathematical formulas available for testing ICR. The simplest 
option calculates percent agreement, determining how many times coders selected the 
same coding option compared to the total number of coding judgments the coders made. 
Under the circumstance of two coders making equal numbers of coding decisions, per- 
cent agreement is also called Holsti’s formula for ICR. More complex formulas used to 
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calculate ICR take into account the fact that coders might actually agree by chance. The 
fewer values a variable has, the more the likelihood that coders could agree not in a mean- 
ingful way but just by virtue of chance. The formulas that take chance agreement into 
account are thought of as more rigorous, and, in fact, reviewers for many Communication 
journals will insist on the use of one of these formulas in order to support publication of 
a content analysis study. 

How do we know whether two or more coders have come to sufficient agreement when 
making coding decisions? Formal ICR tests are conducted and reported with a score that 
ranges from .00 (not reliable at all) to 1.00 (completely reliable). Clearly, higher numbers 
are better, and yet there is no single, universally agreed upon standard for acceptable ICR 
results. Neuendorf (2017) reviewed various established practices for minimally acceptable 
ICR results. She summarized, “It’s clear from a review of the work on reliability that agree- 
ment reliability coefficients that account for chance (e.g., Cohen’s kappa) of .80 or greater 
would be acceptable to all, .60 or greater would be acceptable in most situations, and below 
that, there exists disagreement” (p 168). You can see, then, that although there is no need for 
coders to agree 100% of the time, if their agreement ranges below the 60% mark for the more 
rigorous formulas that account for chance agreement, we will call into question whether the 
study was reliable. 

Arguably, the most versatile test of ICR that accounts for chance agreement is Krippen- 
dorffs alpha. Unlike other formulas, Krippendorff’s alpha can be used with any number of 
two or more coders and can be used for variables across all of the different levels of measure- 
ment. Krippendorffs alpha also takes into account the “magnitude of the misses” (Neuendorf, 
2017, p. 177). The formula factors in, for instance, when Coder 1 selects a value of 1 on a 
scale of 1 to 5 and Coder 2 selects a 2, compared to when Coder 1 selects a 1 and Coder 2 
selects a 5. 

Andrew Hayes collaborated with Klaus Krippendorff (2007) to make a downloadable 
macro to add the capacity to run the alpha test to the statistical software package SPSS, and 
that macro is available for free online. As of this writing, the macro and its corresponding 
manual can be accessed at this website: www.afhayes.com/spss-sas-and-mplus-macros-and- 
code.html. Once the macro is downloaded and saved to your computer, and SPSS is opened, 
the macro is ready to be used. 

It is important to test and report ICR one variable at a time for every variable in the study. 
To use the macro, a new SPSS file must be made for each variable to be tested for ICR. In 
the SPSS file, the coders are entered as the variables, and their names or other identification 
occupy the column positions in the data view of the file (Figure 7.5). The number of rows 
will total to the number of coding decisions that the coders made. In the cells, you would 
enter the values that each coder used to code each variable. In the example here, three cod- 
ers coded ten units. All three agreed on how to code the second and fourth unit. But for all 
other units, at least one of the three coders did not agree with the others. Keep in mind that 
you would have to create a separate file like you see here for each of the variables coded in the 
study to use the SPSS macro to test for ICR. 

Other tests for ICR that take chance agreement into account include Cohen’s kappa and 
Scott’s pi. The statistical software package SPSS allows for the testing of Cohen’s kappa, which 
is an advantage of choosing this formula. Yet, Cohen’s kappa applies only to nominal level 
variables. There are variations on kappa, however, for ordinal measures and for more than two 
coders. Scott’s pi is another example of a chance-corrected ICR formula that is used with 
nominal level data. 

Finally, some advocate for the use of an intra-class correlation coefficient (ICC) to test 
for intercoder reliability, and this is also a function available within SPSS. Used with interval 
or ratio level data, the ICC shows the degree to which the judgments of pairs of coders are 
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associated with each other. The critique of this measure and others like it, however, is that 
testing for the association between two coders’ judgments is not the same thing as testing for 
agreement. Two coders could agree 0% of the time, but their codes could be 100% associated, 
for instance, such as if Coder 1’s values were never the same as Coder 2’s, but they differed 
by the same increment. Lin’s concordance correlation coefficient (Lin, 1989) adjusts for this 
problem and therefore is likely to be a better choice. 

Best practices for ICR include: 


1. Providing an ICR result for each variable rather than just an overall average for ICR 
results. This is necessary so that readers can see which variables were more or less reli- 
ably coded and so that the presence of a variable with weak ICR results in a study is 
not masked by other, stronger variables. In other words, if you simply report that Coder 
A and Coder B agreed at an average rate of 89% across all of the variables, there is no 
way of knowing whether some variables used to calculate that average were very low and 
others were very high. The very low ones may not be sufficiently trustworthy. 

2. Reporting clearly the sample size of the content used to calculate ICR as well as how that 
sample was chosen. 

3. Employing coders who are not authors of the study, if possible, so that coders are not 
aware of the hypotheses, conceptual foundations, and other prior aspects of the study. Such 
awareness may make coders biased, either intentionally or not, in their coding decisions. 


Just like all best practices, of course, there is considerable variation in the degree to which 
such sage advice is actually put into practice. Lovejoy, Watson, Lacy, and Riffe (2016) studied 
randomly selected issues of three major Communication journals spanning 30 years and iden- 
tified 672 content analyses. Among those content analyses, “Only 24.9% of all study articles 
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reported a reliability coefficient for each variable” (p. 1148). Yet, in two of the three journals 
examined, the number of published content analyses reporting a reliability coefficient for each 
variable increased significantly over time. Researchers are getting better, therefore, at meeting 
the best practice of reporting ICR variable by variable in those journals. 

Failure to achieve sufficient intercoder reliability can be attributed either to issues with the 
coding or issues with the coders. If the issues are with the coding, of course, the researcher 
can refine, revise, or otherwise adjust the operational definitions of the variables in the cod- 
ing scheme and then try again to establish intercoder reliability. One such adjustment might 
be reducing the number of categories used to code a variable. Here you may recall the point 
we made in the beginning of the chapter about how it is easier for multiple coders to agree 
that something is 1 = not violent at all, 2 = somewhat violent, or 3 = violent compared to 
1 = not violent at all, 2 = a bit violent, 3 = moderately violent, 4 = quite violent, 5 = very 
violent. If the researcher has operationalized the variable with five options but is having 
trouble reaching an acceptable level of ICR, those five options can be collapsed into three 
and the researcher can try again to see if coders can now reliably agree. In rare cases, variables 
may have to be dropped entirely if even multiple attempts at operationalization still result in 
lack of coder agreement. Sometimes coders themselves may not take sufficient time or care 
with the coding or may develop fatigue while performing the coding, resulting in low inter- 
coder reliability. Or perhaps multiple coders have differing understandings of concepts and 
variables due to differing cultural backgrounds. Additional training is likely needed in these 
situations, and, once again, in rare circumstances, perhaps a coder will need to be dropped 
from the project. 


Research in Action 


Dr. Martha Lauzen is founder and executive director of the Center for the Study 
of Women in Television and Film and professor in the Television, Film, and New 
Media program at San Diego State University. The Center for the Study of Women in 
Television and Film produces Boxed In, an annual report that studies links between the 
number of women working in key positions in the production of television content 
(as producers, directors, writers, etc.) and the ways in which women are depicted on 
TV. The Center also produces a report called It’s a Man’s (Celluloid) World that does the 
same for film. 

We believe the work of the Center for the Study of Women in Television and Film 
is an excellent example of the use of quantitative methods for social justice for two key 
(and interrelated) reasons. 

First, Dr. Lauzen and her colleagues take extra measures to connect the observations 
of media content they make through content analysis to the range of gender identi- 
ties of those who are in key decision-making roles that help to shape that content. To 
make TV and film gender depictions more just and equitable, there is a need for more 
gender diversity among the people who create TV and film content. The quantitative 
data produced by Dr. Lauzen and colleagues show how depictions often fall short of 
being just and equitable when behind-the-scenes decision makers are predominantly 
cisgender men. 
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Second, the ability for the data produced by quantitative researchers to be put into 
the hands of those who can make change is also vitally important. The Center for the 
Study of Women in Television and Film has advisory board members that include a 
number of Hollywood creative professionals, including Mara Brock Akil (a television 
writer and producer), Allison Anders (a film and television writer and director), Cari 
Beauchamp (an author and documentary producer), Madelyn Hammond (the president 
ofa marketing company specializing in entertainment media), Kathy Najimy (an actress, 
writer, producer, director, and activist), Robin Swicord (a writer, director, and pro- 
ducer), and Nia Vardalos (a writer, producer, and actor). By cultivating and maintaining 
relationships with media industry professionals, Dr. Lauzen and colleagues have a direct 
channel for making change. 

The Center’s website uses language that relates directly to the idea we have put forth 
in this book about the power of numbers: “Through its research and programs, the 
Center provides the raw material necessary to change the behind-the-scenes and on- 
screen gender dynamics in the film and television industries, integrating more women 
into vitally important storytelling roles.” 

https://womenintvfilm.sdsu.edu/. 
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Digital and Interactive Media 


Digital and interactive media forms like the internet, social media, and video games represent 
both opportunities and challenges for content analysis. One challenge is the dynamic nature 
of digital and interactive media. How can one claim to say something reliable about media 
content if that content changes based on the ways in which users interface with it? Video 
games are a prime example of interactive media that can be difficult to sustain reliable sys- 
tematic content analysis (Schmierbach, 2009). If two people play Fortnite for twenty minutes 
each, for instance, they will see different content depending on how they navigate their avatar 
in the virtual world. 

Researchers have undertaken various strategies to attempt to address this problem when 
applying content analysis methodology to interactive game content. One strategy is to recruit 
multiple people to play the game and record their gameplay to code content features. Another 
is to study recorded game play sampled from YouTube or other online videos. Yet even these 
analyses produce observations that are shaped by the decisions of the players themselves. 
Nonetheless, these are feasible options that certainly show some of what is likely to be encoun- 
tered in the game. 

The massive amounts of data produced by social media and other information sources 
in the digital age also pose particular opportunities and obstacles for scholars interested in 
the systematic documentation of communication content. YouTube is, of course, a prime 
contemporary outlet for blogs, web video, and other user-produced content. In 2015, You- 
Tube CEO Susan Wojcicki reported that an estimated 400 videos are uploaded to YouTube 
every minute (Brouwer, 2015). Given its rapidly shifting nature, therefore, researchers study- 
ing YouTube content, regardless of the methods they use and the size of their sample, can 
take only a tiny and fleeting snapshot of the entire landscape. Nonetheless, a cursory glance 
through recently published articles shows a number of content analyses of YouTube content, 
ranging from those pertaining to health topics such as obesity (Yoo & Kim, 2012) or organ 
transplants (Tian, 2010), to those addressing issues in the representation of Native Americans 
(Kopacz & Lawton, 2013). Some of these studies examine the content features of YouTube 
videos themselves, whereas others also study the comments associated with those videos. 

The age of “big data” generated by Facebook, Twitter, and other sources and the corre- 
sponding development of computer algorithms to both capture and code those data are 
important developments. Computers can be programmed to (1) capture content coming from 
the “firehose” of big data, (2) identify the occurrence (or co-occurrence) of particular words 
and phrases within huge stores of data, and (3) make inferences based on the features of the 
content about underlying factors such as the opinions and feelings expressed in the content or 
the power dynamics or personality traits of the people who produced the data (Grimmer & 
Stewart, 2013; Lacy, Watson, Riffe, & Lovejoy, 2015; Zamith & Lewis, 2015). Automated 
content analysis is an increasingly popular approach to the study of communication content. 

There are at least three main approaches used in automated coding of communication con- 
tent to date. For some automated coding, researchers use a dictionary-based approach— 
employing predefined dictionaries containing words and phrases that belong to different 
linguistic or topical categories—to allow the computer to identify the presence of particular 
features of content. The widely used Linguistic Inquiry and Word Count (LIWC) program 
(Pennebaker, Boyd, Jordan, & Blackburn, 2015) includes approximately 90 word categories 
that connote emotional tones, sentiment types, topics, and other information. Performing a 
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“sentiment analysis” on social media data, for instance, automatically looks for positively and 
negatively valanced words in the communication content to determine whether the social 
media users who produced that data discussed a topic positively or negatively. LIWC is able to 
scan text and make inferences from the text as to the motives, personality traits, power status, 
and identities of the person who created the text. Although there are many fruitful applica- 
tions of LWIC or other dictionary-based programs, they are somewhat limited in their ability 
to reflect the nuances of the entire set of meanings contained in the texts. 

The second approach, supervised content analysis, uses supervised machine learning 
to train computers to recognize patterns in content based on learning from data that have 
already been classified into patterns (Su et al., 2017). In this type of computational approach, 
researchers develop a codebook just as they would in a conventional content analysis and 
human coders manually code data. The results from the manual coding are used to train com- 
puters to perform the coding. This approach has been applied to analyses studying sentiments 
expressed in communication content, frames used in news content, and other topics. 

A third approach is unsupervised automated content analysis, in which computers 
are not trained with predefined rules, dictionaries, or codebooks but rather look inductively 
for semantic patterns to identify underlying structures of content. Based on the assump- 
tion that words or phrases that are close to one another in meaning tend to appear close 
together in text, semantic network approaches determine whether words co-occur (or study how 
closely words appear in proximity to one another) within a set of content. A variation on this 
approach studies whether different concepts are used in similar contexts or in similar ways 
(such as that they co-occur with other words) regardless of whether they appear in close prox- 
imity to one another (Arendt & Karadas, 2017). Also within this category, researchers use topic 
modeling approaches to discover underlying themes, relationships, and meanings within texts 
through the discovery of similar semantic structures and patterns contained within those texts. 

In order to make these computational approaches more concrete, let’s take a closer look at 
an example. Arendt and Karadas (2017) used a technique called Contrast Analysis of Seman- 
tic Similarity (or CASS, Holtzman, Schott, Jones, Balota, & Yarkoni, 2011) to study a social 
justice-related topic, whether newspaper coverage makes associations between Muslims and 
violence or terror. To conduct their study, the authors gathered every news article within a 
particular range of dates published in any German newspaper archived in the LexisNexis data- 
base that contained the words Islam, or Christianity, or both, resulting in articles that totaled 
over 2.5 million words. Applying the automated coding CASS approach, Arendt and Karadas’ 
(2017) study “revealed substantial mediated associations between Islam-related concepts and 
violence (e.g., ‘Koran + violence’), terror (e.g., ‘Islam + terror’), dehumanization (e.g., ‘Mus- 
lims + animal-related terms’), and general negativity (valence)” (p. 107). Within these millions 
of words, the words Muslim, Islam, Koran, mosque, and imam were associated with violence, 
terror, and negatively valenced words more than were the words Christianity, Christian, Bible, 
church, or priest. 


Critiques of Big Data Analysis and Critical Approaches to 
Datafication 


At the same time that digital technologies and big data analytics represent opportunities for 
researchers as well as new affordances for users, there are concerns about living in an age in 
which businesses, governments, and other institutions are increasingly engaging in “datafica- 
tion.” So many aspects of daily life are now turned into data, and that data is often tracked, 
stored, monitored, and used not just for research but for commercial and/or surveillance 
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Figure 7.7 Big data analysis 


Source: https://www.shutterstock.com/image-vector/set-flat-line-color-banners-design-305497067 


purposes. The rising role of big data analytics has been the subject of critique on both meth- 
odological and ethical grounds. 

Using automated or computational approaches to research, for instance, has been cri- 
tiqued on two main points: the validity of the data derived and the privacy of users’ data. The 
question of how valid findings from automated coding may be—how well they reflect, accu- 
rately and thoroughly, the full range of meanings contained in communication content—is a 
common methodological critique. Automated coding is more reliable than human coding as 
well as less time-consuming. Indeed, programmed to apply a set of rules for classifying con- 
tent to a set of messages, the computer will do so quickly and will do so the same way every 
time, making it 100% reliable. Yet, there are certainly limits to the validity of how and what 
computers can code, leading many experts to suggest that, at least to date, they cannot fully 
replace the nuanced, complex reasoning that human coders can devote to content analysis 
(Grimmer & Stewart, 2013; Lacy et al., 2015; Zamith & Lewis, 2015). Some research com- 
bines computational approaches with traditional qualitative or quantitative human coding to 
ensure that the context and interpretation of messages circulated via social media are better 
understood. Sarah J. Jackson and Brooke Foucault-Welles (2016), for instance, used both 
social network analysis and discourse analysis to study the ways in which messages circulated 
via Twitter following the death of Michael Brown, the Black man shot and killed by a police 
officer in Ferguson, Missouri. They discuss their collaborative research in the Scholar Spot- 
light in Chapter 10. 

Second, as we mentioned at the beginning of this chapter, critique has emerged about 
the privacy of data analyzed using automated coding approaches. Typically, the social media 
or other internet users whose data are examined using computational approaches have not 
given consent for their data to be the subject of research, even if they have posted those data 
to public platforms. Furthermore, although researchers typically take measures to protect the 
anonymity of social networking users whose data are examined, there have been breaches 
associated with data of this type, in which people’s identities, information, or accounts have 
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been exposed. With technology advancing in ways that are likely to facilitate the gathering of 
big data through automated means, concerns about how to collect and analyze data ethically 
will continue to be important. 

Scholars in critical data studies are concerned with the ways in which big data and the 
processes and practices of datafication can be used to reinforce rather than dismantle social 
inequalities (Boyd & Crawford, 2012; Crawford, Gray, & Miltner, 2014). Those engaged in 
critical data studies are asking important questions about whose data are being used and for 
what purposes, and, more generally, who stands to benefit and who stands to lose in datafi- 
cation. They remind us that even if computers can be 100% reliable in describing data, the 
data itself is never neutral but rather is produced in particular contexts with particular social, 
economic, and political conditions. Among the social justice principles interrogated in critical 
data studies are: 


e Privacy and surveillance. Just by living in a digital society, we are subject to scrutiny and 
study from the digital footprints we leave in our daily interactions with technology (Lyon, 
2018). The potential for surveillance and loss of privacy to lead to harm is more likely 
among more vulnerable groups (Gangadharan, 2017). 

e Inequality and discrimination. There are power imbalances between the masses of people 
who produce big data and the small groups who control and use big data. In the book, 
Algorithms of oppression: How search engines reinforce racism (2018), for instance, Saftya Umoja 
Noble discusses how online searches are optimized by those with both the technical skill 
and the economic means to do so, leaving the less powerful further disadvantaged. Pre- 
dictive policing, using algorithms to predict where crime may occur, is another example 
of a computational practice that has been critiqued for fueling racism and classism. 


In fact, issues of privacy and surveillance and inequality and discrimination are intertwined. 
Underlying each are the capitalist logics embedded in datafication that can both exclude and 
exploit (Couldry & Mejias, 2019). Virginia Eubanks’ (2018) book, Automating inequality: How 
high-tech tools profile, police, and punish the poor, for example, shows the ways in which poor 
people are surveilled and harmed when decisions about welfare, child protection, and public 
housing are made through the use of algorithms. 

The issues of ethics and the social justice principles raised in critical data studies are impor- 
tant for researchers using algorithmic and computational approaches in their research as well as 
for all researchers and citizens, more generally. Being aware of and responsive to these issues is 
precisely the sort of critical consciousness that we are encouraging in this book. Researchers 
using computational approaches to study communication content have the capacity to shed 
light on the ways that digital media have failed to produce a more just and equitable society. 
They can also help facilitate data activism efforts that empower marginalized groups using the 
same tools. 


Summary 


In this chapter, we have outlined the ways in which content analyses are designed and 
carried out, pointed to the purposes of using this research method, and highlighted the 
strengths and weaknesses of content analysis as a research tool. As with other quantitative 
methods, validity and reliability are the twin goals, and special attention must be devoted 
to sampling and the measurement of concepts and variables. Unique to content analy- 
sis research is the need to establish acceptable levels of agreement among multiple cod- 
ers. Additional concerns that take on particular importance in content analysis have been 
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introduced in this chapter, as well, such as how units of analysis are determined and where 
and how to access communication content. 

Although we have focused on traditional coding carried out by humans, we have seen that 
computer learning and algorithms are used in some studies of communication content, and 
we have touched more generally upon the ways that interactive media forms, overall, have 
provided challenges and opportunities for the content analyst. Indeed, the broad technologi- 
cal changes that allow for both the production of big data and the use of machine learning 
and algorithmic coding to study big data pose ongoing issues related to unequal access and 
differential harm stemming from loss of privacy and enhanced surveillance. 

Content analysis can expose the limits of the representation of social groups, issues, and 
events in media or other communication content. Conversely, content analysis can shine light 
on positive depictions and progressive developments in the ways that topics, events, and peo- 
ple are represented. With carefully constructed samples designed to generalize to the popu- 
lation from which they are selected, thoughtful conceptualization and operationalization of 
concepts, and rigorous testing of intercoder reliability, there is strong potential for this method 
to be used toward a more just and equitable society. 


Discussion Questions and Activities 


1. What topic related to social justice, diversity, and/or inclusion do you think would be 
good to study using content analysis research? Why? 

2. Come up with two or three variables that you could study using content analysis meth- 
odology, identifying conceptual definitions for each. Next, determine how you would 
operationally define each of the variables. To develop operational definitions, imagine 
you are giving detailed instructions to coders regarding how to code the variables. 

3. Do some exploring online to find electronic databases, archives, or other resources you 
might use to access communication content in order to perform a content analysis study. 
What did you find? What types of content analysis questions might you be able to explore 
within that content? 

4. How do you think content analysis can be used to work toward social justice? What are 
the opportunities of as well as the barriers to doing so? 
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8 Calculations and Complexity 


As we have established, quantitative researchers collect data in the form of numbers. Once 
data are collected, researchers explore the values and properties of the data and examine 
numerical relationships within the data. Statistical analysis is used to identify numerical pat- 
terns and calculate relationships between and among variables, applying mathematical prin- 
ciples and procedures to analyze and interpret what the data suggest about the phenomenon 
being studied. 

In many ways, the statistical analysis of quantitative data is rather straightforward. Parsi- 
mony is a goal of quantitative research, which means that statistical analyses should provide 
the simplest possible approach that offers the best explanation of the phenomenon being 
investigated. There are established sets of rules and processes laid out for quantitative research- 
ers to guide them in summarizing the features of the data that they have collected. There are 
widely accepted standards on which to base interpretations of whether the patterns discovered 
should be considered trustworthy contributions to knowledge. Computer programs take on 
the burden of the calculations themselves, leaving the researcher to simply input the data, 
select specific statistical tests, and make sense of the results. In the next chapter, we will show 
you how to run statistical analyses and interpret output from SPSS, Statistical Package for 
Social Sciences, a commonly used tool. 

Statistics are a vitally important way to understand the world and to intervene in injustices 
and inequalities. Statistics are powerful tools for those working toward social justice, as they 
use scientific and mathematical processes to make broad and defensible claims. The scientific 
and mathematical techniques behind statistical analysis give those claims authority. And the 
broadness of the claims—statistics are not about one but instead are about many—show just 
how pervasive a phenomenon is. Claims made through statistical analysis are not easy to dis- 
miss, given these properties. 

Yet, there are complexities involved, as well. As we have discussed in Chapter 1, human 
behavior, societal conditions, and cultural norms, often the foci of quantitative Communica- 
tion researchers, are incredibly complex. Statistical relationships may appear to give a clear- 
cut answer to a research question that explores patterns within this broad, multifaceted, and 
sometimes quite complex backdrop. Yet, although they certainly tell us something of value, 
there is inevitably more to the story than the conclusions reached through statistical analysis 
of quantitative data. In fact, it is crucial to remember that there are humans behind statistics— 
human experiences, human views, human hopes, and sometimes human pain. In studying 
a social issue or topic using the tools of statistical analysis, perhaps we can examine the phe- 
nomenon scientifically and see aspects of the phenomenon we might otherwise miss. But we 
must bring ethics and compassion to this work, as well. 

We have called this chapter “Calculations and Complexity” to convey the main processes 
and principles used to analyze and interpret quantitative data—the “calculations” —while still 


Calculations and Complexity 183 


keeping in mind the multifaceted human phenomena underlying those calculations—the 
“complexity.” Quantitative researchers must do their best to paint a picture of the phenom- 
enon they are studying that is as thorough and complete as possible in addition to being accu- 
rate and credible. Recalling the twin goals of validity and reliability, quantitative researchers 
must simultaneously allow for nuance and meaning even as they test for and communicate 
broad patterns within a relatively large data set. Indeed, quantitative researchers guided by a 
concern for social justice have a unique responsibility to honor the differences among the 
people providing data or the texts being studied and help make clear the context in which the 
data are gathered and interpreted. 


Individual Data Points Aggregated Together 


Typically (but not always) in quantitative research, data are drawn individually (from each per- 
son in a survey or experiment, or each bit of communication content in a content analysis) 
and then aggregated—combined to form groups of individual data points about whom/ 
which conclusions are drawn. Aggregation is akin to providing data in summary form, and 
summarizing is necessary when you are dealing with large data sets, as is typically the case in 
quantitative research. We introduced this idea in Chapter 1. 

With survey methodology, for instance, rather than reporting on each individual’s answer to 
each individual item on the questionnaire, we aggregate or combine those answers to simplify 
the data and find patterns. With experiments, the members of the control group typically 
form an aggregate, as do members of the treatment group, to facilitate comparisons between 
the two. Within those control and treatment groups, we might also examine subgroups, as 
you saw in Chapter 6. With content analysis, data can be aggregated in many different ways, 
such as by grouping individual observations of texts by genre, by topic, by date, or by some 
other organizing principle. 

In a quantitative study, it is impractical and inadvisable to report the exact responses of each 
of the many research participants (or, in the case of content analysis, each of the many obser- 
vations of texts) as the findings. Therefore, we use data aggregated from individual values to 
report patterns. Aggregation is a defensible and even inevitable aspect of quantitative research. 
In communicating the results of a study, the researcher must summarize findings, finding ways 
of reducing the raw data into its most important features. This is the case in nearly any study, 
but it is especially necessary for quantitative research since the sample sizes tend to be larger 
than in qualitative research. 

Figure 8.1 is an example of what a raw data file might look like in an Excel spreadsheet, 
with just 11 fictional research participants. Recall that to subject data to statistical analysis, all 
data have to be numerical and in a format that can be read by a computer, which is reflected 
in this Excel file. Yet, rows and rows of data like these would be almost impossible to make 
sense of without some sort of summary and interpretation of that summary. The researcher 
has to decide how to simplify and summarize rather than merely presenting the raw data as the 
study’s results. In fact, most studies have many more variables compared to this example (the 
ones in the current example can be found in the column positions) and many more research 
participants. So, the raw data in most studies are likely to be even less understandable on their 
own than this example here. 

Aggregation is also a huge strength of quantitative research. Recall from Chapter 4 that the 
purpose of quantitative inquiry is to identify patterns that are larger than a single experience 
or that are broadly representative of a whole host of individuals or texts. In order to make 
these more far-reaching observations that are vital to our understanding of the world around 
us, we must engage in aggregation. Observations determined from an aggregate cannot be 
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Figure 8.1 Raw data file 


Source: Stephen Warren 


written off as “just one person’s opinion” or “just a few cherry-picked examples.” Rather, 
such observations are larger in size and scope and—especially if the sample is carefully created, 
as we learned in Chapter 4—they carry with them a degree of authority in representing broad 
populations. 

Yet, by focusing on aggregates, we also run the risk of failing to notice cases that run coun- 
ter to the primary patterns identified in the data. Imagine an experiment in which subgroups 
defined by the self-reported gender identity of the participant are examined, for instance. If 
women and men in the treatment group do, indeed, differ in their response to the experi- 
mental stimulus, this difference is based on the aggregation of the sample by gender. Yet, it is 
almost guaranteed that not every woman differed from every man in the experiment. Rather, 
the women and the men differed in general, as groups. Therefore, the decision to aggregate 
by gender identity allows us to know about differences between groups but not necessarily 
within groups. In other words, our example would illuminate differences between (or among, 
if there are more than two groups) groups defined by gender identity. But it would take a dif- 
ferent decision about how to aggregate in order to shed light on differences within the group 
of self-identified men or within the group of self-identified women. Perhaps, for instance, you 
would want to aggregate the self-identified men by race to now test for differences across men 
with different racial identities. Or you might want to aggregate the self-identified women by 
sexual identity to now test for differences across women with different sexual identities. This 
technique will get you closer to an intersectional approach to understanding identity that we 
discussed as an important framework in social justice research. 

If one is concerned with voice, inclusion, and power, the use of the aggregate rather than 
the individual as the focus of analysis can also reinforce dominant positions, even if that is not 
the intention of the researcher. In other words, if quantitative research focuses on the major- 
ity—through reporting, for instance, that 73% of respondents on a given college campus have 
not experienced discrimination, we may lose track of the 27% who have. Their experiences 
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are not the majority but in this case are likely to be the most important aspect of the phenom- 
enon to bring to light through the research. 

So, how do you reconcile the use of aggregation and identification of general patterns 
as defining features of quantitative research while also making sure you are attending to the 
richness and nuance of human experience and human difference? First, the researcher must 
maintain curiosity and openness about the people or texts being studied. When there is 
an indication of a pattern such as a gender difference in an experiment, for instance, the 
researcher should ask why that is the case, or for whom it is the case, or under what condi- 
tions is that the case. The researcher can design a follow-up study to that original experiment 
that further explores the gender difference found, placing the study of the gender identity 
of participants at the center of the analysis. If that follow-up study is quantitative, it will still 
use aggregated groups to identify patterns—again, such is the stuff of quantitative research. 
Yet, that new study will add depth to our understanding of the topic. Furthermore, relying 
on research participants who enjoy dominant positions in society is likely to be insufficient 
and perhaps even problematic in terms of further marginalizing or minoritizing groups. As 
explained in Chapter 4, more strategic sampling strategies are likely to be needed to ensure 
that small or minoritized groups are given voice in quantitative research. 

The knowledge that is gathered in quantitative research is incremental, with each new find- 
ing and each new study building on the last. We can learn a great deal from different aggre- 
gates of people and texts, and each new inquiry furthers the richness of our understanding. 
As other “quantitative criticalists” have pointed out (and as discussed in Chapter 1), bringing 
quantitative research methods to social justice efforts will rest on the desire of the researcher 
to imbue every study with meaning, context, and nuanced interpretation. 


Spotlight Scholar: Dr. Dana Mastro 


Dana Mastro (PhD, Michigan State University) is Professor in the Department of Com- 
munication at the University of California Santa Barbara, where she studies media and 
racial and ethnic stereotyping. Dr. Mastro researches patterns in media content pertain- 
ing to the representation of racial and ethnic minorities as well as the influence of media 
on views of self and others among both non-dominant (mostly Latinx) and majority 
White groups. The author of numerous influential research articles, Dr. Mastro was 
named a fellow of the International Communication Association in 2018. She is also 
the former chair of the Mass Communication division of the National Communication 
Association and the recipient of a 2019 Distinguished Teaching Award at her university. 


Q: What role do you think statistics can have in promoting social justice efforts? Do you think 
statistics are an important way to work toward social change? If so, why? 

Dr. Mastro: If the goal of social justice efforts is not only to address disparities in access 
to fundamental human rights and social privileges but also to confront inequi- 
ties in terms of opportunity to achieve these ends, then statistics are an essential 
partner in such efforts. Statistics give us the context to understand the scope and 
severity of human events and experiences. They allow us to identify inequality in 
resources, power, and representation as well the obstacles to achieving parity in 
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Figure 8.2 Scholar Spotlight: Dr. Dana Mastro 


these outcomes. Perhaps most importantly, they are instrumental in developing 
strategies to address these issues. 

Q: What do you think are the limits or challenges of using statistical analysis as a means of 
engaging in social justice efforts? Are there any pitfalls that researchers should avoid? 

Dr. Mastro: One pitfall to the use of statistics in social justice and advocacy efforts 
is the potential to lose sight of the individuals and experiences represented in 
these numbers. To highlight just two examples: It is necessary to create cat- 
egories when using statistics, and it is common and even essential to “drop” or 
“remove” outliers to move forward with valid statistical analyses. In both cases, 
best practices in statistics can result in losing sight of social context, social com- 
plexity, and meaningful human conditions. This means that researchers must be 
vigilant in giving voice to who and what their statistics represent (and fail to 
represent). 

Q: Your own work is on media and racial/ethnic stereotyping. Why do you use quantitative 
methods /statistical analysis to study those topics? What do you think those quantitative 
and statistical tools bring? 

Dr. Mastro: My orientation toward quantitative approaches stems from my belief that 
demonstrating causal links between media use and race-related outcomes strength- 
ens efforts aimed at social, institutional, and policy change. My personal identity 
and experiences throughout my life called attention to these relationships and 
prompted my desire to study these issues. Using quantitative approaches allows for 
the identification of systematic patterns in the representations of diverse groups in 
the media and gives us the ability to examine how these characterizations affect 
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outcomes (both favorable and unfavorable), ranging from cognitions and emo- 
tions to policy preferences to behaviors toward different groups in society. 

Q: Learning statistics can be difficult. What advice would you give to a graduate student in 
an Introduction to Quantitative Research Methods course who is encountering statistics for 
the first time? 

Dr. Mastro: From my perspective, statistics are best learned when being applied to 
issues about which we are invested and passionate. This helps bring clarity to 
techniques and processes that can otherwise seem abstract and inaccessible. In 
other words, statistics become demystified when they illuminate the phenomenon 
we care about as scholars. That said, the only way to truly learn statistics is to use 
statistics, so taking every opportunity to work with multiple faculty and collabora- 
tors is essential to solidifying and expanding one’s knowledge and ability. 


Different Types of Statistics 


Before we begin a discussion of statistical analyses that you may perform on the data you have 
gathered in a quantitative research project to summarize your findings, which is the subject of 
Chapter 9, it is necessary first to define key terms. The word statistics has multiple meanings. 
It is used to describe the subfield of mathematics that researchers use to make sense of numeri- 
cal data. As we discovered in Chapter 4, the word also implies that the researcher has drawn 
a sample from a population and is summarizing the properties of and relationships between 
variables within that sample. (Recall that a sample statistic is used to estimate a population 
parameter.) The use we are referencing in this section of the chapter is statistics as “numerical 
summaries of data obtained by measurement and computation” (Vogt, 1999, p. 278). 

Univariate statistics examine the values of one variable at a time. For example, if you 
collect information about respondents’ age in a survey, when it comes time to write up your 
findings, you could report on just that variable alone by stating the average age of the people 
who responded to the survey. Bivariate statistics examine relationships between two vari- 
ables. Are younger survey respondents more likely to express a desire for a more diverse cast 
of characters in a video game compared to older respondents? To answer that question with 
data gathered in a quantitative study, you would see whether respondents’ age is related to 
their opinion about the characters that appear as options when playing video games. This calls 
for two variables to be at play in a statistical test. Finally, multivariate statistics examine 
relationships among three or more variables. Maybe you also want to test whether respond- 
ents’ use of video games, in addition to their age, might be linked to their views on the desire 
for a more diverse set of playable options. You now have three variables in the analysis: age, 
amount of video game use, and views on whether video games should have more diverse 
playable characters. Any time there are more than two variables at play in statistical analysis, 
the researcher is engaged in multivariate statistics. 

Univariate analyses are used to describe, whereas bivariate and multivariate analyses are 
used to explain. Univariate analysis describes the properties of the values that comprise each 
of the variables in the study. Looking at one variable at a time cannot allow the researcher 
to know whether or how two variables relate to each other. To test that, the researcher must 
simultaneously examine two or more variables (using bivariate or multivariate statistical analy- 
sis), so that they can determine whether scores or values on one variable help explain scores 
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or values on another variable. If there is a statistical relationship between age and support for 
same-sex marriage, for instance, knowing a research participant’s age will help explain the 
position they are likely to hold on same-sex marriage. Or, if there is a statistical relationship 
between being watching a documentary film about Islamophobia and opinions on restrict- 
ing travel to the U.S. for individuals from predominantly Muslim countries, then knowing 
whether the research participant saw the documentary will help explain the position they are 
likely to hold on the travel restrictions. In general, when writing up results in most quantita- 
tive studies, researchers typically begin by communicating the patterns apparent in the sample 
variable by variable via univariate statistical analysis before moving on to bivariate or multi- 
variate analyses. 


Hypothesis Testing 


Hypotheses are the bedrock of quantitative research. As you learned in Chapter 1, quantita- 
tive research takes a deductive approach to inquiry, beginning with a theory (in this case, a 
prediction made in the form of a hypothesis) and then making observations to see whether 
that theory is supported or refuted. By nature, therefore, quantitative research requires an 
extensive review of existing studies and building of logic to form a basis for the predictions 
made in hypotheses. Contrast this approach with qualitative research methods that are more 
inductive, making observations first and then fitting a theory to what is observed. For this 
reason, then, hypotheses are necessary in quantitative research in order to organize and help 
structure the study. 

Before expressing each hypothesis, there should be sufficient theory and literature review 
to lead logically and convincingly to the prediction made in the hypothesis. Yes, a hypothesis 
is widely known as an “educated guess,” but the “educated” part of that oft-used phrase is 
important. A hypothesis is not merely a guess. It is a persuasive, logically sound prediction 
made on the basis of prior knowledge. The prior knowledge, in turn, is based on previously 
conducted studies and/or on previously existing theories. 

Indeed, the idea of collecting data first without a sense of what you are predicting and 
why you are predicting it and then engaging in rather haphazard explorations of relationships 
within that data is called “fishing” in quantitative research and is frowned upon. If you fish 
around for relationships within the quantitative data you collect, you run the risk of find- 
ing something that relates but having no logical or reasonable explanation for why it relates. 
Quantitative research inquiries should be well grounded in theory and prior research so that 
the predictions made are logical and defensible. 

In fact, hypotheses should be written before data collection. It is widely considered mis- 
leading and even unethical to collect data, perform statistical analyses to look for patterns, 
and then write down predictions after the fact, given that you already know whether those 
predictions have been supported. There are some internet sites where you can pre-register 
your hypotheses, reporting them to a third-party source, so that it is abundantly clear that 
they were developed before the data were collected and analyzed. Pre-registration of a study 
or a set of hypotheses is part of an open science movement that seeks to make the research 
process more transparent and reduce the likelihood that researchers will retrofit their predic- 
tions based on patterns they have already discovered in the data. If you are interested, you can 
find out more at the Center for Open Science website. We also discuss open science in more 
depth in Chapter 12. 

A research hypothesis is defined as “a statement of (or conjecture about) the relation- 
ships among the variables that a researcher intends to study” (Vogt, 1999, p. 134). Because 
hypotheses are about how two or more variables relate, then, to test them the researcher uses 
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bivariate or multivariate approaches to statistical analysis. To return to a previous example, the 
researcher may predict that the younger the respondent, the more likely they are to endorse 
the view that video games should have more diverse characters. The prediction is that age will 
be statistically related to views of games’ characters. 

The null hypothesis is the opposite of the research hypothesis. Rather than making a 
prediction about what the researcher expects to find, a prediction is made about what the 
researcher expects not to find. This type of hypothesis is called a null hypothesis because the 
researcher expects that the data observed will nullify the prediction (Figure 8.3). As Vogt 
(1999) points out, “In accumulating evidence that the null hypothesis is false, the researcher 
indirectly demonstrates that the variables are related” (p. 195, emphasis in original). The null 
hypothesis will figure prominently in our subsequent section on statistical significance. 

The relationship predicted between variables in a research hypothesis can be either nondi- 
rectional or directional. A nondirectional research hypothesis predicts that variables will 
be related, but it does not specify the direction of that relationship. For example, one group 
may be predicted to differ from another group on some variable of interest in a study, but 
the researcher does not hypothesize which group will score higher (or lower) on that vari- 
able of interest compared to the other group. Conversely, and true to its name, a directional 
research hypothesis predicts the direction of the relationship between variables, and there- 
fore is more specific and detailed than a non-directional hypothesis. For example, rather than 
simply saying that amount of news consumption will be related to fear of crime, we might 
say as television viewing increases, fear of crime also increases. In this case, then, we have 


| am what is 
The default, the status quo 
| am already accepted, can only be rejected 


The burden of proof is on the alternative 


| am the null hypothesis 


freshspectrum.com 


Figure 8.4 The null hypothesis assumes no relationship between or among variables 


Source: Cartoon drawn by Chris Lysy of FreshSpectrum.com 
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predicted the direction of the relationship, adding greater specificity and precision to our 
hypothesis. In general, if there is good reason to predict a direction for a relationship based 
on prior studies and the application of theory, it is preferable to use a directional research 
hypothesis rather than a non-directional one, since a directional hypothesis is more precise. 

Here are some examples of directional hypotheses from our own research. In the latter two 
examples, multiple predictions are made within the same hypothesis statement and noted as 
a, b, and c. 


* Students will show less favorable attitudes about the presence of gender stereotypes in 
advertising after participating in the media literacy education program compared to 
before (Sekarasih, Scharrer, Olson, Onut, & Lanthorn, 2019). 

e Exposure to negatively framed Obama news stories as compared to positively framed sto- 
ries will result in (a) greater anti-Black affect and (b) increased stereotypical perceptions 
of African Americans, which will (c) lead to increased symbolic racist beliefs (Ramasu- 
bramanian & Martinez, 2017). 

e — Ethnic media use will be positively correlated with (a) ethnic pride, (b) ethnic performance, 
and (c) self-esteem for Indian Americans (Ramasubramanian, Doshi, & Saleem, 2017). 


Here is an example of a non-directional hypothesis: 


e Violent video game exposure will correlate with the endorsement of “traditional” mas- 
culine gender roles (Blackburn & Scharrer, 2018). 


The most essential consideration in the crafting of a hypothesis is that it must be testable/fal- 
sifiable. This means that the observations made in the study must be able to be used to either 
support or refute the hypothesis. The hypothesis must be sufficiently specific, concrete, 
and clear in order to be tested. Note that we usually say that the data/observations “support” 
rather than “prove” and “refute” rather than “disprove” hypotheses. In other words, when the 
data match the prediction made in a hypothesis, we say the hypothesis was supported; we do 
not say that the hypothesis was proven. When the data do not match the prediction made 
in a hypothesis, we say that hypothesis was either not supported or refuted, rather than say- 
ing the hypothesis was disproven. Researchers typically steer clear of suggesting something 
has been proven or disproven beyond the shadow of a doubt. Given that hypothesis testing is 
based on probability theory (as you will see a bit later in this chapter), even with a supported 
hypothesis, there is always some degree of chance that the pattern found in the sample used 
in the study does not exist in the wider population. That is the chief reason why quantitative 
researchers use such cautious language. 


Research in Action 


Crafting hypotheses for your research 
The content 
e Hypotheses are typically phrased in the form of a statement about the relationship 


between two (or sometimes more) variables. The relationship referenced in that 
statement will either be supported or refuted by the data. 
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e Example: “The perception that using social media for politics is impactful will 
be positively associated with using social media for political expression” (Kwak 
et al., 2018, p. 201). 


e Hypotheses should have at least two variables. This will ensure that you have a 
hypothesis that is testable using bivariate or multivariate statistical analysis. 


e Example, in which the two variables at play are in bold: “The perception 
that using social media for politics is impactful will be positively associ- 
ated with using social media for political expression” (Kwak et al., 2018, 
p. 201). 


¢ But do not put too many variables in the same hypothesis! This is because you 
want hypotheses to either be supported or refuted by patterns you discover in the 
data, but not both supported and refuted simultaneously. If you have too many pre- 
dictions all at once, some predictions may be supported and some refuted, leading 
to “partial support” for the hypothesis and potential confusion. 

e The predictions expressed in hypotheses often use one of two main forms: 


e Form A: One or more group will differ from one or more other group. Or, 
better yet, one or more group will score higher (or lower) on some variable 
than one or more other group. 

e Example: “H1: Instagram users who view positive posts from strangers expe- 
rience a) more negative affect and b) less positive affect than Instagram users 
who view neutral posts from strangers or no posts” (de Vries, Moller, Wieringa, 
Eigenraam, & Hamelink., 2019, p. 227). 

e Form B: Two variables will be positively associated with each other (as values 
in one variable increase, values in the other variable also increase) OR two 
variables will be negatively associated with each other (as values in one variable 
increase, values in the other variable decrease). 

¢ Example: “H1b: There will be a positive relationship between pro-attitudinal 
selective exposure about Latino culture and social media political expression 
about Latino culture” (Velasquez, Montgomery, & Hall, 2019, p. 151). 


The format and style 


e Hypothesis are numbered and sometimes abbreviated with an “H” for hypoth- 
esis, like, H1, H2, H3. 

e There is no standard or ideal number of hypotheses that a single quantitative 
research study may include, but most seem to have between two and six. 

e If you have multiple predictions that are closely related to one another, a useful 
approach is to label them H1a, H1b, H1c, etc. 


e Example: “H1a: The perception that using social media for politics is impact- 
ful will be positively associated with using social media for political expres- 
sion... . H1b: The perception that using social media for politics is easy will 
be positively associated with using social media for political expression” (Kwak 
et al., 2018, pp. 201-202). 


e Hypotheses should be concise. After you draft a hypothesis, read it back to see if 
there are any words you can cut out while still retaining the meaning. 
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e Hypotheses should be clear. Use the same terms to describe variables in the 
hypotheses that you have used elsewhere in your quantitative research paper. 


Sources: 


de Vries, D. A., Moller, A. M., Wieringa, M. S., Eigenraam, A. W., & Hamelink, K. (2019). Social 
comparison as the thief of joy: Emotional consequences of viewing strangers’ Instagram posts. Media 


Psychology, 21(2), 222-245. https://doi.org/10.1080/15213269.2016.1267647 


Kwak, N., Lane, D. S., Weeks, B. E., Kim, D. H., Lee, S. S., & Bachleda, S. (2018). Perceptions 
of social media for politics: Testing the slacktivism hypothesis. Human Communication Research, 44, 
197-221. https://doi.org/10.1093/hcr/hqx008 


Velasquez, A., Montgomery, G., & Hall, J. A. (2018). Ethnic minorities’ social media political 
use: How ingroup identification, selective exposure, and collective efficacy shape social media 
political expression. Journal of Computer-Mediated Communication, 24, 147-164. doi:10.1093/ 
jemc/zmz007 


When there is insufficient prior knowledge to justify a prediction or when prior findings 
related to the prediction contradict each other, researchers use a research question rather 
than a hypothesis. The only substantive difference between a hypothesis and a research ques- 
tion in quantitative research is that a hypothesis appears in the form of a statement and a 
research question appears in the form of a question. For instance, where a hypothesis might 
predict that two variables will be related to each other, a research question asks whether they 
will be related to each other. Research questions in quantitative research should not be broad 
and open-ended, as might be utilized in inductive, qualitative research. Although they are 
phrased as a question rather than a statement, ideally, they should still be as precise and specific 
as possible. Here is an example in which there are actually two research questions in one, as 
is indicated by the notations (a) and (b). You can see that although the relationships between 
variables are phrased as a question in which the outcome is uncertain, the research question 
still spells out a specific relationship between variables to be tested in the research. 


RQ1: Does a positive relationship exist between group self-definition and social media 
political expression about (a) Latino culture and (b) immigration? (Velasquez et al., 2019, 
p. 150). 


As discussed in Chapter 2, in the steps of a quantitative research project, hypotheses typically 
appear after the literature review and before the methods section. They follow the literature 
review because, once again, hypotheses should stem logically and convincingly from the prior 
studies and theories reviewed. And they precede the methods section because in methods the 
researcher discusses how the study is designed in order to collect data to test the hypotheses. In 
fact, before you begin data collection, you will want to closely compare the methods of your 
study to your hypotheses. Do the methods you have developed allow for your hypotheses to 
be tested? In other words, if you conduct your study the way your methods section indicates, 
will you have the data you need to either support or refute your hypotheses? Given that the 
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literature and theory are laid out first in quantitative research and then a detailed roadmap 
about how the study is designed is presented in the methods section—all before any data is 
collected—you can pause at this stage to ensure your predictions and proposed methodology 
connect well to each other. 


Error, Uncertainty, and Rejection: Oh, My! 


Given that (1) hypotheses are based on a certain degree of conjecture, (2) there are limitations 
to the samples in which hypotheses are tested (as we saw in Chapter 4), and (3) topics studied 
in social science are quite complex, some hypotheses will fail to be supported by the results 
of the statistical analyses used to test them. In other words, researchers’ predictions could turn 
out to be off the mark. When you read published quantitative studies, you will notice that it is 
quite common to have one or more hypotheses rejected. If this happens to you (and it will!), 
do not despair. You are in good company. 

A rejected hypothesis could mean that the prediction itself was flawed in some way. The 
researcher thought there was sufficient evidence from prior research to make a prediction 
about what would transpire in the specific context of a new study. Yet perhaps there was 
something different about the new context that meant the previous research did not apply. 
For instance, maybe prior research found that social media use was associated with body 
image disturbance among college student participants. On the basis of that prior evidence, 
the researcher hypothesizes in a new study that the same pattern will be found among an 
older, general adult population. If that hypothesis is rejected based on the findings in the data, 
the researcher may speculate that perhaps older individuals are less vulnerable to such a social 
media influence. (In yet another example of how cautious quantitative researchers typically 
are, once again, we would merely speculate about this rather than state it as true. More research 
that tests this new prediction—that older adults are less vulnerable to an influence of social 
media use on their body image—would be needed to make a bolder claim.) 

On the other hand, another reason why hypotheses may not be supported is because of 
the error or uncertainty that inevitably takes place when you draw a sample to study from a 
larger population. In this case, then, the prediction was not necessarily flawed, but the sam- 
pling process resulted in the researcher failing to find the predicted relationship between or 
among variables. In the case of social media and body image disturbance, for instance, even if 
the data point to no relationship between those two variables within a sample of the general 
adult population, it could be the sample that is flawed (recall the content of Chapter 4) rather 
than the theory that formed the basis of the prediction. In other words, if we were somehow 
able to look at the entire population of adults, we might find that their social media use does, 
indeed, relate to their body image. But, in that original study, given that samples do not always 
accurately reflect the population from which they are drawn, we simply did not find this. 

Due to the uncertainties involved in sampling, two types of errors may take place 
(Figure 8.5). Type I error refers to a situation in which the researcher rejects the null hypoth- 
esis on the basis of the evidence found in the study, but in the population the null hypothesis 
was true. This occurs, for instance, when the researcher concludes that two or more variables 
are likely to be related to each other in the population because they were found to relate to 
each other in the sample. But, alas, although the two or more variables were related to each 
other in the sample, they were not actually related in the population. Type II error is used to 
describe the scenario in which the researcher decides—based on the data in the sample—that 
the null hypothesis is true, but in reality in the population, the null hypothesis is false. In this 
case, there was a relationship between two or more variables in the larger population, but the 
researcher did not find it in the current study and therefore concluded that it was not likely 
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Figure 8.5 Type I and type II errors 


Source: Stephen Warren 


to exist. (The same logic applies in medical testing, for example, in which a patient may get a 
“false negative” or a “false positive” test.) 

Of course, researchers would like to avoid making either Type I or Type II errors. Thank- 
fully, Type I error can be estimated mathematically, and therefore the degree of risk of mak- 
ing this sort of mistake is known in research. The researcher can determine how much risk 
of Type I error is acceptable and will estimate the amount of this error incurred in a study 
through tests of statistical significance. We will discuss statistical significance at length in 
the subsequent section of this chapter. On the other hand, the researcher typically has no 
way of knowing they are making Type II error, since the population parameter is unknown. 
We can try to avoid Type II error, though, by making sure the sample size is large enough to 
detect a relationship between variables that occur in the population. You may want to revisit 
the discussion in Chapter 4 about why larger samples are—for the most part—more likely to 
reflect the population compared to smaller samples. 

Perhaps an example would help illustrate why and how this matters in research. Imagine 
that a researcher is using content analysis to study whether people with disabilities are confined 
to minor character status rather than main character status in a sample of primetime television 
content. Based on prior research findings, the researcher hypothesizes that characters with dis- 
abilities will be more likely than characters without disabilities to be minor rather than major 
characters. The observations they make in their new study on the topic, however, do not 
support this hypothesis. The results find no relationship between whether the character had a 
disability and whether the character was in a minor role in the program. The researcher might 
be tempted to say, “Fantastic. Television content has progressed in its depiction of people with 
disabilities, no longer relegating them to minor character status.” Yet, alas, it is possible that a 
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different sample would have led the researcher to a different conclusion. This is why the quan- 
titative researcher is trained to use cautionary language about conclusions drawn from samples. 
In this case, the researcher might report, “In this study, no relationship was found between the 
character’s (dis)ability status and their major or minor role in the program. This finding could 
indicate a positive change in television depictions compared to prior research findings. How- 
ever, additional evidence from future research is necessary to support this conclusion.” 


Making an Inference 


One more distinction between types of statistical analyses is important to introduce, as well. 
Descriptive statistics are used to summarize observations within a sample, whereas inferen- 
tial statistics are used to infer whether an observation found in a sample is likely to also occur 
in a larger population. Can we confidently estimate a population parameter from a sample 
statistic? Inferential statistics are used to answer that question. 

Making a claim regarding the likelihood that a result found in a sample is operating in the 
population at large is an important pursuit for quantitative researchers, and therefore inferen- 
tial statistics are frequently used. However, sometimes the researcher does not intend to gen- 
eralize from a sample to a population, or sometimes the manner in which the researcher drew 
the sample does not allow for such generalizations. In this case, descriptive statistics are used to 
describe the sample without attempting to infer to a larger population. Therefore, descriptive 
statistics are limited to describing or summarizing the data collected without attempting to 
determine whether the same patterns found in such description and summarizing are likely 
to be operating in a wider population. 

What can we infer about the population parameter given the evidence we have found in 
the sample? We know from Chapter 4 that an inevitable amount of uncertainty and impreci- 
sion occurs when samples stand in for populations, even if those samples are randomly drawn. 
There are two main approaches for answering the question of what we can infer about the 
population on the basis of data observed. We will focus our discussion on one, Null Hypoth- 
esis Significance Testing (NHST) since it is dominant in most social sciences (Ortega & 
Navarrete, 2017). We invite the reader to learn about the other approach, Bayesian statis- 
tics, using a different source. 

When a researcher conducts statistical analysis in the NHST framework, the math calcu- 
lated by the statistical software will produce two results for each hypothesis: the finding for 
the statistical test and the finding that indicates statistical significance. In the next chapter, 
we will walk you through the steps involved in deriving both of those numbers in SPSS and 
interpreting what is found. But for now, it’s important to simply know this: The first finding 
in the statistical results shows the mathematical relationship between the variables. The second 
finding indicates the probability of obtaining the first number (the relationship between the 
variables found in the study) if the null hypothesis were true (Haller & Krauss, 2002). The 
null hypothesis assumes the relationship between the two variables in the population is zero: 
It assumes they are not related. How likely is it to obtain the relationship between variables 
apparent in the data—that’s the result observed in the study, and it is derived from the statisti- 
cal test—if there were no relationship between the variables in the population? That is the 
question that the result of the test for statistical significance answers. 


Introducing the Probability Value 


A researcher obtains a result in a study that seems to indicate a relationship between vari- 
ables. But that relationship between variables may not be present in the population. Statistical 
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significance, then, is a way of measuring the likelihood of making a Type I error, the likeli- 
hood of rejecting the null hypothesis when, in reality, the null hypothesis is true. Statistical 
significance is expressed as a probability value or p value (with probability answering the 
question of how likely it is) and abbreviated with a lowercase “p.” Since it has to do with prob- 
ability of making a Type I error, the probability value can be interpreted in the same manner 
as an odds ratio. The smaller the p value, the smaller the odds of rejecting the null hypothesis 
when it is, in fact, true in the population. 

But once again, how small must the odds of making a Type I error be to reject the null 
and, instead, claim support for the research hypothesis? Given that we can never completely 
eliminate the possibility of making a Type I error when we sample from a population, how do 
we decide what likelihood of making a Type I error is acceptable? Many decades ago, standard 
levels of statistical significance were established and, despite controversy and debate over 
whether quantitative researchers should still be applying them, those same standard levels of 
statistical significance are widely used and broadly accepted today. 

The long-standing practice in quantitative research is to make the decision about whether 
to reject the null hypothesis based on the significance level obtained in the statistical results as 
compared against one of three standards: .05, .01, and .001. Therefore, you will typically see 
statistical results reported as p < .05, p < .01, or p < .001. 


e For p < .05, that means that the likelihood of finding the relationship between variables 
that was observed in the study when there is no such relationship in the population is less 
than 5 in 100. If the researcher obtains a p value within this range (< or = .05), they can 
reject the null hypothesis. 

e For p < .01, that means that the likelihood of finding the relationship between variables 
that was observed in the study when there is no such relationship in the population is less 
than 1 in 100. If the researcher obtains a p value within this range (< or = .01), they can 
reject the null hypothesis. 

e For p < .001, that means that the likelihood of finding the relationship between variables 
that was observed in the study when there is no such relationship in the population is less 
than 1 in 1,000. If the researcher obtains a p value within this range (< or = .001), they 
can reject the null hypothesis. 


You can see, then, that the odds of making a Type I error if we have a result that has a p level 
of .001 are lower compared to a result with a p level of .01. And a result with a p level of .01, 
in turn, runs less risk of Type I error than a p level of .05. The smaller the p, the more 
plausible the research hypothesis is compared to the null hypothesis. In other words, 
the smaller the p level, the more unlikely it would be to obtain the results we have in the 
study showing a relationship between variables when there was no relationship between the 
variables in the population. 

Long ago, it was determined that the .05 significance level is just about the highest amount 
of uncertainty and imprecision that we can live with as social scientists. That is why there 
is still a long-standing tradition of comparing specific p level results found in a quantitative 
study against this standard. Therefore—and this is important!—researchers have to say that any 
statistical result they obtain in which the probability level is greater than .05 is not sufficient 
evidence on which to reject the null hypothesis and claim support for the research hypothesis. 
If p is greater than .05, we must say that the research hypothesis is rejected. This means that 
the odds are just too great that what was found in the sample is not operating in the wider 
population. When writing up a quantitative research study, the convention is to report such a 
result as non-significant. Non-significant is often abbreviated as ns. 
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Figure 8.6 Hypothesis testing 


Source: Stephen Warren 


One last, but also important, point before we move on. Even though the size of the p 
value indicates the plausibility of the research hypothesis rather than the null, it does not 
indicate the strength of the relationship between the variables. In other words, a result that 
is significant at the p < .001 level does not mean that the variables are more strongly related 
compared to a result that is significant at the p < .01 or p < .05 level. It simply means that 
the odds of observing the relationship found in the sample when that relationship is not 
occurring in the population are smaller if p < .001, slightly larger if p < .01, and slightly 
larger still at p < .05. 

Let us illustrate this entire section of the chapter with an example. In the next chapter, we 
will introduce the Pearson’s correlation test, which is a test that measures association between 
two variables: As the values of one variable go up or down, do the values on the other vari- 
able do the same? In a survey, for instance, as the values that are used to measure respondents’ 
education increase, do the values that are used to measure their income also increase? If so, we 
say the two variables are correlated, more education is associated with higher income. Using 
SPSS to test whether education and income are correlated, the findings might look like this: 
Pearson’s correlation = .38, p < .01. The .38 is the first number we have been referring to in 
this discussion. It shows the relationship between education and income. (Again, we will zero 
in on what a .38 Pearson correlation means in the next chapter.) The second number is the p 
level. It shows that the likelihood of observing a correlation between education and income 
of .38 as was observed in the sample when there is no relationship between education and 
income in the population is less than 1 out of 100. If the researcher had hypothesized that 
education and income would be correlated, then, they could claim support for that hypothesis 
on the basis of these results. 
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Statistical significance is a very important principle in statistical analysis, and is quite fre- 
quently misunderstood. To summarize, then, it might be helpful to break down the logic of 
tests of statistical significance in a quantitative research study into 10 points. 


Point 1: A research hypothesis predicts that a relationship between variables will be found 
in the data. (A research hypothesis is usually referred to simply as a hypothesis when 
writing up a quantitative research study.) 

Point 2: A study is conducted to see if the relationship between variables predicted in the 
hypothesis is observed in the data. 

Point 3: A statistical test (like Pearson’s correlation in our example) describes the relation- 
ship between variables observed in the data. 

Point 4: Because the data from the study usually stem from a sample, there is always the 
possibility that the study’s findings will not accurately reflect the population. 

Point 5: The null hypothesis is the converse of the hypothesis. It states that there will be no 
relationship between variables. 

Point 6: The test for statistical significance answers the following question: How likely is it 
that the relationship between variables would be observed in the data if that relationship 
were not present in the population? 

Point 7: If there is a sufficiently low probability of obtaining the relationship between 
variables observed in the data if that relationship does not also occur in the population 
(i.e., making a Type I error), the null hypothesis is rejected and the research hypothesis 
is supported. 

Point 8: The conventional standard for how low the probability of making a Type I error 
must be to decide the research hypothesis is more plausible than the null hypothesis is 
.05 or lower. 

Point 9: A probability value of .05 or lower means that the likelihood of finding the rela- 
tionship between variables that was observed in the data when there is no such relation- 
ship in the population is less than or equal to 5 in 100. 

Point 10: Although the threshold for statistical significance of .05 has been contested, it 
is still widely in place as the standard in the social sciences for claiming support for the 
research hypothesis. 


What Explains Statistical Significance? 


Two main factors determine statistical significance: Sample size and the strength of the rela- 
tionship between the variables. You are more likely to find a statistically significant relation- 
ship between variables when you have a large sample (given its advantage in reflecting the 
population) and when the variables are strongly rather than weakly related. If variables are 
strongly related, even a relatively small sample is likely to reflect this relationship. In contrast, 
if variables are weakly related, a larger sample would be needed to detect the relationship. 

Earlier in the chapter, we discussed an example of a research study that found no support 
for a hypothesis that characters with disabilities would be more likely to be presented in minor 
roles in primetime TV compared to characters without disabilities. Imagine that in this study 
the researcher had a sample size of 110 characters, and this was the distribution for the vari- 
ables of interest (remembering than n means sample size): 


Character with a disability Character without a disability 
Minor character n=17 n= 41 
Major character n= 13 n= 39 
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Now imagine that the researcher studied this phenomenon in a sample of 1,100 characters 
rather than just 110. Even if there was the same pattern in the distribution, as is shown below, 
it is possible that the finding is now statistically significant, simply because the sample size is 
larger and better able to detect a rather weak relationship between characters’ ability status and 
their major/minor role. 


Character with a disability Character without a disability 
Minor character n= 170 n= 410 
Major character n = 130 n = 390 


This reminds us, then, that rejecting a hypothesis that tests a relationship between variables in 
a small sample on the basis of the result for statistical significance does not mean that the two 
variables are guaranteed to be unrelated in the population. It may mean that the researcher 
needs a larger sample to reduce sampling error and be more likely to capture the population 
parameter. 

Furthermore, on the flip side of the coin, given the realities about how statistical signifi- 
cance is calculated, just because two variables are found to be related in a statistically sig- 
nificant manner in a large sample does not necessarily mean that the finding is important or 
even meaningful. In large samples, it is not difficult to find a relationship that is statistically 
significant, but that does not necessarily mean that there is real-world importance for that 
finding in the larger population. The first author of this book remembers working on a large 
sample with co-authors and jokingly saying, “I just sneezed and it was statistically significant!” 
as a silly way of communicating the role of sample size in determining statistical significance. 
“Statistically significant” is not the same as important or “significant” in the colloquial sense. 


Some Finer Points Regarding Statistical Significance 


There is a bit of wiggle room sometimes allowable when the p level is just barely greater than 
.05 for any given result. The researcher can make the case that the relationship between the 
variables “approaches” statistical significance if the p level is .055 (which would round to .06) 
and slightly larger. Sometimes, you may see claims that results approach statistical significance 
with p levels all the way up to .10. If you do get a result with a p level within this range (.06 
to .10), however, you should interpret that result with an extra dose of caution. You should be 
aware, as well, that even cautious conclusions based on these results may not be acceptable to 
some readers of the study, including some anonymous reviewers or journal editors who help 
determine whether a study will be published. 

The odds of making a Type I error also grow with the number of statistical tests run. 
Therefore, one important way to control for Type I error is to be conservative in the num- 
ber of statistical tests run in a study. In order to be particularly rigorous and hold themselves 
up to a high standard, researchers may also adjust the p level to be lower if they have a large 
number of statistical tests to run. In one common strategy to correct for this problem known 
as the Bonferroni method, the typical .05 threshold for statistical significance is divided by the 
number of statistical tests run. So, if your hypotheses call for 20 tests to be run, the Bonfer- 
roni adjusted significance level would be .0025 (.05 divided by 20). In order to claim a result 
is statistically significant with this Bonferroni corrected significance level, the p value would 
have to be less than or equal to .0025. 

Some researchers choose to disregard the .05, .01, and .001 tradition of reporting of statisti- 
cal significance altogether. Rather, these researchers opt to report the exact p level associated 
with a research finding. In fact, computer programs like SPSS that calculate statistical tests and 
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statistical significance do, indeed, report the exact probability level in the output they provide. 
Instead of converting an exact p value to the smallest possible of the three traditional com- 
parison levels, some researchers will simply report the exact p value itself. In this case, then, 
there is no need to use the less than sign (as in p < .05). If you read a quantitative article that 
says, for instance, p = .03, then you know this researcher has elected to report exact p levels. 
In this case, then, you would interpret p = .03 by concluding that the likelihood of obtaining 
this finding when it is not also operating in the population is 3 in 100. 

Technically, the researcher should not use inferential statistics when the sample they are 
drawing from is a nonprobability sample. If random selection was not used to create the 
sample—the defining characteristic of nonprobability samples—then, strictly speaking, the 
mathematical principles of inferential statistics do not apply. However, in practice in Commu- 
nication and other fields, inferential statistics continue to be used for nonprobability samples 
when those samples, regardless of whether they are randomly selected, are meant to stand in 
for larger groups. If you read published quantitative studies in Communication as well as in 
other fields, you will find plenty of examples of authors using inferential statistics to analyze 
patterns found in nonprobability samples. 

Costs associated with obtaining probability samples are rising, and even probability sam- 
ples often suffer from individuals dropping out of the study or otherwise not responding. 
Therefore, many researchers are making a case for the ability of the researcher to infer from 
nonprobability samples to a wider population. Andrew W. Mercer (a senior research meth- 
odologist at the Pew Research Center) and colleagues, for instance, suggest that researchers 
attend to the composition of the sample (including whether it contains all possible subgroups) 
and measure multiple variables that might factor in to the topics examined (Mercer, Kreu- 
ter, Keeter, & Stuart, 2017). For example, if you are interested in a topic pertaining to U.S. 
political party formation, you would want to ensure members of all political parties are pre- 
sent in the sample. If you were interested in knowing whether political party identification 
helps explain support for immigration policies, you would also want to include variables that 
measure other potential predictors of such support. When researchers take these steps, Mercer 
and others contend, they may, indeed, make unbiased estimates of the population based on a 
nonprobability sample. 


Summary 


Central to the logic and design of quantitative research is the aggregation of data, a necessary 
step in summarizing patterns observed. We might lose a bit of the nuance we would find if 
we explored a smaller set of data more in depth, but we gain a great deal in being able to 
point systematically and scientifically to an aggregated observation in a large sample. We do 
so because, by nature, those observations are larger than any single experience and because 
they cannot be critiqued as having been “cherry picked” in order to make a particular point. 

Indeed, the principle of aggregation works together with the a priori nature of hypothesis 
testing to frontload the design of a quantitative research study with predictions derived from 
prior knowledge and the procedures that will be used to obtain data and examine patterns and 
properties in the data. Hypotheses should be clear, precise, and as succinct as possible so that 
they can be refuted or supported through statistical analysis. Even with such clear, precise, and 
succinct hypotheses, however, there is always the possibility of erroneous claims made about 
whether findings within a sample are likely to accurately reflect the population in the form 
of Type I and Type II error. 

Statistics are used to examine the data to look at properties of variables, one at a time 
(univariate statistics) and to test for differences or relationships between and among variables 
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(bivariate and multivariate statistics). Statistical significance is reported alongside the results of 
bivariate and multivariate tests to state the likelihood of observing the relationship found if 
it does not exist in the wider population. As such, the researcher makes inferences from the 
sample to the population. 

Learning statistics is sort of like learning a new language. There is a steep learning curve 
at first, with many new terms and techniques to master, as you have seen in this chapter. But 
just like learning a language, learning statistics is incremental, with new knowledge building 
on top of a growing foundation. Emphasizing the context in which statistical observations 
are drawn and the meaning behind the patterns discovered is a crucial part of undertaking 
quantitative research to contribute to social justice efforts. 


Discussion Questions and Activities 


1. The claims made through statistical analysis always have caveats because statistics is based 
on probabilities and uncertainties. How or why do you think quantitative researchers can 
still communicate their findings in a way that adds importantly to the topic in question, 
despite these caveats? 

2. What do you think are the advantages and disadvantages of making observations based on 
aggregated data in quantitative research? 

3. What do you think of the long-standing tradition of treating p < .05 as a critical threshold 
for statistical significance? Do you agree or disagree with that tradition? Why? 

4. Imagine that you are a researcher who is interested in whether time spent watching 
make-up and beauty tutorials on YouTube is related to satisfaction (or lack thereof) with 
one’s appearance among teens. Write a hypothesis that would test a prediction about that 
topic. 

5. Imagine that you are a researcher who is interested in whether responses to an organiza- 
tion’s statement about racial injustice and anti-Black racism following the Black Lives 
Matter protests in summer 2020 differed among employees with different racial identities. 
Write a hypothesis that would test a prediction about that topic. 
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9 Statistical Analysis 


From Principles to Practice 


In the preceding chapter, we introduced the logic and language involved in the statistical 
analysis of quantitative data. This current chapter extends that logic and language into more 
practical and precise questions about (1) how to run statistical tests on quantitative data and 
(2) how to make sense of what you find out from those tests. This chapter, then, translates the 
more abstract discussions of Chapter 8 into concrete steps to put those principles into practice 
when conducting a quantitative research study. 

Given that the math involved in statistical tests may raise some anxieties, we have approached 
this chapter with what we hope you will find to be a clear and easy to understand style. We 
have also focused on a handful of the most foundational statistical tests, even though that is 
really just the beginning of statistical understanding. Within the limited number of statistical 
tests that we introduce in this chapter, we have narrowed our discussion further to only the 
most important components of those tests. Finally, in keeping with the rest of this book, we 
continue to emphasize interpretation and not just mechanics: not just how to run statistical 
analyses but how to interpret what you find. 

The primary tool that you need to conduct statistical analyses is a statistical software pro- 
gram. Although there are a number of options available (such as STATA, SAS, and R), in 
this chapter, we focus on SPSS, Statistical Package for Social Sciences. Acquired by IBM 
in 2009, SPSS is used in many fields within and outside the social sciences. SPSS features 
easy-to-use pull-down menus (or the user can choose to write syntax) and supports a wide 
range of statistical analyses. The addition of program macros and “plugins” further extends 
the flexibility of SPSS (including its ability to interact with Python, a computer program- 
ming language increasingly used in research). SPSS is not free, but there is a student discount 
available, and many Communication departments are likely to have the software installed on 
university computers. 

We will detail a number of processes involved in entering data, analyzing data, and writ- 
ing up results from that data analysis in this chapter, using a sample data file to illustrate these 
processes. The sample data file is from our research and can be accessed at www.drsrivi.com 
or on this book’s Routledge.com webpage. From either of those websites, you can download 
the data file and follow along with the processes described in the rest of this chapter, as well 
as do some exploring on your own. The data set features demographic variables (like gender, 
race, ethnicity, region of the country where the respondent resides, sexual orientation), politi- 
cal ideology (on a scale from very conservative to very liberal), television and video game use 
variables, and variables that measure respondents’ views of masculine gender roles and norms. 
The measures in the sample data set are from a survey study of 18- to 25-year-old adults in 
the United States. 
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Steps Involved in Creating and Analyzing Quantitative Data 


Each of these steps will be explored in full in this chapter, but it is important to lay them all 
out before we begin so that you can see the entire process and plan accordingly in your own 
research. 


10. 


11. 


Data collection—lIf you are collecting your own data (rather than using an existing data 
set that you download from another source), the first step is to use the research design 
approaches we have outlined for experiments, surveys, or content analyses to collect the 
data. 

Data entry—Next, you must get the data into computer-readable form, typically by 
entering or uploading the data into SPSS or any other program you are using. As you will 
see in the next section, this may mean manually entering your data, variable by variable, 
case by case. Or, if you have collected data online, it may be as simple as importing your 
data directly into SPSS. 

Naming variables—In order to keep track of your data, you must name each variable 
so that it can be easily identified and readily understood. 

Determining variable type—A key step at this stage in the process is figuring out 
whether your variables are measured at the nominal, ordinal, interval, or ratio levels. This 
is necessary because there are a large number of statistical tests available to researchers, yet 
each test requires that the data have particular properties. Chief among those properties is 
the level of measurement. 

Determining variable values—As you learned in Chapter 3, variables can take on a 
number of values. These are the numbers you assign to each possible option associated 
with the variable, such as 1 = yes and 2 = no. 

Data cleaning—Before beginning data analysis in earnest, the researcher studies the data 
to see whether there are any mistakes made with data entry or any other errors. If your 
variable was measured as 1 = yes and 2 = no, for instance, and you see a 3 entered, this 
is an error and needs to be resolved. If you keep the 3 in and run statistical analyses, the 
analyses will be thrown off by the mistaken value, and they will be inaccurate. 

Data reduction—Sometimes, we use multiple measures that can be combined before 
beginning statistical analyses. This is called data reduction because it takes multiple vari- 
ables and reduces them down to one single or a few variables. It is a way to ensure that 
the statistical analyses are as straightforward and parsimonious as possible. 

Recoding variables—SPSS and other statistical software will allow you to manipulate 
the manner in which your variables are measured. For instance, if you used five categories 
to measure agreement with some statement in a survey, you could collapse that range of 
responses into just three categories. This process is called recoding. 

Creating indices—For some measures, multiple items are used to measure the same 
concept. These individual measures can often be combined into one single measure by 
creating an index (also known as a scale). Creating an index is a specific form of data 
reduction and again is helpful so that your data analysis is as parsimonious as possible and 
is easy to understand. 

Descriptive statistics—Before hypotheses are tested, first the data are summarized vari- 
able by variable. These analyses describe the data and are thus called descriptive statistics. 
Determining statistical tests—After descriptive statistics are run, hypotheses are tested 
and research questions addressed using one or more statistical tests that combine variables 
to study relationships between them. The researcher comes up with a data analysis plan 
to decide which test(s) should be run (once again, based in large part on the properties of 
the data). 
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12. Conducting analyses—This is the step in which the data analysis plan is put into place. 
One or more statistical tests are run to test hypotheses and answer research questions. 

13. Interpreting results—Statistical software produces output associated with each statisti- 
cal test that has been run. The researcher must analyze that output to figure out what the 
result of the test(s) is, interpreting the output to make sense of the findings. 

14. Writing up results—The final step in statistical analysis is to write up the results, put- 
ting the results into both a narrative form and, often, in the form of tables and/or graphs. 
In this step, it is important to be clear, accurate, and comprehensive in terms of the details 
of the output shared with readers so that they can be informed about the results. Relat- 
edly, in this step, the researcher communicates whether hypotheses were supported or 
refuted and provides an answer to research questions that had been posed. 


Creating a Data File in SPSS 


Before data can be analyzed, of course, you have to have data in the SPSS data editor. When 
you first launch SPSS, it will give you an empty spreadsheet that you can use to input your 
own data, and we will describe how to do so in detail a bit later in this chapter. To access the 
sample data set provided to readers of this book rather than entering your own data, however, 
you would follow the commands seen in the screenshot from SPSS (Figure 9.1.1). This set of 
commands (File Open— Data) will allow you to open any data set that you have saved on 
your computer. First, you download the sample data set we have provided on either www. 
drsrivi.com or on this book’s Routledge.com webpage and save it to your computer. Then 
you use this set of commands to open the data set in SPSS on your own computer. 

Also visible (but not highlighted) in Figure 9.1.1 is the Import data function. Some 
researchers create their data files in another program such as Excel, and SPSS easily imports 
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Figure 9.1.1 The Open File command in SPSS 


Source: Stephen Warren 
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Excel files into its data editor using the Import data function. If you have used a survey plat- 
form like SurveyMonkey, Google Forms, or Qualtrics, you can download data into an Excel 
file and then upload the data from that Excel file to SPSS. Some of those resources also allow 
the researcher to download the data directly into SPSS. When you import data, however, you 
will still have to use Variable View (described next) to enter the properties of the data you 
imported so that SPSS can read the data you have uploaded. 

The SPSS data editor works as a basic spreadsheet, and you can toggle back and forth from 
Data View to Variable View at the bottom left of the screen. Variable View is the screen 
that you need to input your data, with cells that will allow you to name each variable that you 
have in your study and specify the values and properties of each. Once you have done this step 
(creating variables and specifying their properties) and the data are entered, you can toggle 
back to Data View to look at the data set as a whole, now composed of all the variables you 
created in Variable View. When creating your variables in Variable View, each variable will 
be in a row position. When you switch back to Variable View, those variables will now be in 
column positions rather than rows. 

Figure 9.1.2 provides a screenshot of Data View, followed by a screenshot of Variable 
View (Figure 9.1.3a), each showing the data from the sample data set provided to readers 
of this book. In these visuals, you can see the variable names that we gave to the variables 
in question (more on that soon!). Many of the variable names are pretty self-explanatory, 
especially when read alongside the label that you can see in the visual. But in case you were 
wondering about some of the less self-explanatory variable names, the MRNI items come 
from the Masculine Roles and Norms Index-Revised, an index created by Ronald Levant 
and colleagues to measure people’s views about social roles and cultural norms associated 
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9.1.2 Data view 


Source: Stephen Warren 
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with masculinity (Levant et al., 2007). That index has multiple subscales, as you can see in 
the screenshots: 


Restrictive Emotionality, which measures views of whether dominant masculine norms favor 
stoicism or lack of emotional expression, especially around stifling emotions that might 
be considered to indicate weakness 

Self-Reliance, which measures views of whether dominant masculine norms favor hav- 
ing know-how about stereotypically “masculine” tasks like home improvement or car 
repairs 

Negativity Toward Sexual Minorities, which measures views of whether dominant masculine 
norms are homophobic or heteronormative 

Avoidance of Femininity, which measures views of whether dominant masculine norms 
include avoiding behavior or roles traditionally associated with femininity 

Importance of Sex, which measures views of whether dominant masculine norms favor 
sexual conquest and/or sexual drive 

Toughness, which measures views of whether dominant masculine norms encourage boys 
and men to be physically tough and unafraid of risks 

Dominance, which measures views of whether dominant masculine norms favor leadership, 
being the boss, taking charge 


The other variables that may not be self-explanatory in the screenshots of the sample data 
set include the ADMI_A, which is a set of measures from the Auburn Differential Mascu- 
linity Index—Aggression (Burk, Burkhart, & Sikorski, 2004) that examine views of whether 
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Figure 9.1.3a Values of a variable 


Source: Stephen Warren 
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aggression and physical violence are favored in views of dominant masculine roles and norms. 
Finally, the variables that have DUM in them are named that because they are dummy vari- 
ables. A dummy variable is a variable that has been simplified to have only two values, with a 
0 assigned to the value that will be considered a reference group and a 1 assigned to all other 
values. We will revisit dummy variables later in this chapter. 

When you are setting up your own data file rather than using the sample data set, under 
Variable View the first step is to name your variables. You can name your variables anything 
you like, as long as the name does not surpass the maximum number of characters, contains 
no spaces, does not duplicate other variable names, and conveys to you the meaning of the 
variable. There is also a field in which you can enter a longer explanation of the variable 
name, without limits to characters and spaces, which SPSS calls a label. 

Under Variable View, you will see a place to indicate the values of the variable 
(Figure 9.1.3b). If you have a variable for which there is a relatively small number of categories 
used to classify data—as is the case for nominal, ordinal, and interval level variables—you use 
this field to indicate the numbers assigned to each category. For example, for each variable 
that measures agreement with a statement using a standard five-point Likert scale, you would 
tell SPSS that 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 
and 5 = strongly agree, if those were the response options you used. (Side note: Although 
you could flip the order and say 1 = strongly agree, 2 = agree, 3 = neither agree nor disagree, 
4 = disagree, 5 = strongly disagree, generally it is better to have higher numbers mean more 
of the variable in question, in this case, more agreement, rather than less. This makes the 
interpretation of the values on that variable easier because it is more intuitive.) Or, to set up 
a variable for political party affiliation, you would indicate that 1 = Democrats, 2 = Repub- 
licans, and 3 = Independents if, indeed, you operationalized that variable with those three 
categories. 


ece @ Example Data.sav (DataSet?) - IBM SPSS Statistics Data Editor 
3 O, r a A 
389 Neo SLIP 4} BE mod 
Name Type Width Deci... Label Values Missing Col... Align Measure Role 


92 MRNLAF3 Numeric 8 
93 MRNI_AF4 Numeric 8 
94 MRNI_AFS Numeric 8 
95 MRNI_AF6 Numeric 8 
96 MRNI_AF7 Numeric 8 
97 MRNIIS1 Numeric 8 
98 MRNI_IS2 Numeric 8 
99 MRNI_IS3 Numeric 8 
100 MRNI_T1 Numeric 8 
101  MRNI_T2 Numeric 8 
102  MRNI_T3 Numeric 8 
103 MRNI_T4 Numeric 8 
104 MRNI_D1 Numeric 8 
105 MRNI_02 Numeric 8 
106 = MRNI_D3 Numeric 8 
107 MRNI_D4 Numeric 8 
108 = MRNI_OS. Numeric 8 
109 MRNI_D6 Numeric 8 
110 MRNI_D7 Numeric 8 
111 ADMI_A1 Numeric 8 
112 ADMI_A2 Numeric 8 
113 ADMI_A3 Numeric 8 
114 ADMI A4 Numeric 8 


Boys should prefer to play with truc... {1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
Aman should prefer watching actio... {1, Strongly Disagree}... None 10 MRight #Scale input 
Men should not wear make-up, cov... {1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
Boys should not throw baseballs lik... (1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
Men should not be interested in tal... {1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
Aman should always be ready for s... {1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
Aman should not turn down sex. {1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
Men should always like to have sex. {1, Strongly Disagree}... None 10 MRight #Scale ‘input 
When the going gets tough, men sh... {1, Strongly Disagree}... None 10 Right #Scale ‘Input 
| think a young man should try to b... {1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
Men should get up to investigate if ... {1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
It is important for a man to take ris... {1, Strongly Disagree}... None 10 MRight #Scale ~ input 
Men should be the leader in any gr... {1, Strongly Disagree}... None 10 MRight #Scale ‘Input 
A man should always be the boss. {1, Strongly Disagre: --- 10 # Scale ‘Input 
Men should make the final decision... {1, Strongly Disagree}... Ê Scale ‘Input 
The President of the U.S. should alw... {1, Strongly Disagree}... Ê Scale ‘Input 
A man should provide the discipline... {1, Strongly Disagree)... Ê Scale ‘Input 
A man should always be the major ... {1, Strongly Disagree}... Ê Scale =~ Input 
In a group, it is up to the men to ge... {1, Strongly Disagree}... # Scale =~ Input 
If another man flirts with a man’s gi... (1, Strongly Disagree}... Ê Scale input 
Sometimes, a man’s got to fight or ... {1, Strongly Disagree}... Ê Scale ~ Input 
If a man made a pass at another ma... {1, Strongly Disagree}... None 10 MRight Z Scale input 
It's okay for a man to use physical v... {1, Stronaly Disaareel... None 10 mRiaht #Scale ~ Input 


Ota View RA 


coocoooeococoscoocoooooooooocso 


IEM SPSS Statistics Processor is ready Unicode:ON 


Figure 9.1.3b SPSS display of values 


Source: Stephen Warren 
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If you have a ratio level variable, which, as you recall, is a variable that can take on a wide 
range of values including a meaningful 0, you do not have to complete this step. That is 
because that variable is a continuous variable, taking on any number of values from the 
minimum found in the sample to the maximum, and the value itself is meaningful. For exam- 
ple, imagine you are an intercultural communication scholar and you asked on a survey: How 
many times a day do you interact with a person from a different country or culture?, with 
respondents filling in a specific number to answer. In the data editor, you can just enter that 
exact number for each value. That is the case for any ratio level variable. 

The screenshots from SPSS show that, under Values, SPSS displays just the first among 
many values for each variable. If you were to click on the ellipsis (. . .) that appears just after 
SPSS tells you what the first value is for a variable, you could see the other values, as well, as 
you can see in Figure 9.1.3b. 

Some of the features of Variable View that you can see in Figure 9.1.4 that we did not yet 
explain, are: 


Numeric: a field to indicate whether the variable is comprised of simple numbers, num- 
bers with a particular order and meaning like dollars or dates, or even letters; note that 
if the variable is in letters you would have to convert those letters to numbers to use the 
variable in statistical analysis 

Width, Decimals, Columns, and Align: all fields that simply have to do with the format 
of the display 

Measure: a field you can use to indicate the level of measurement for your variables 
(nominal, ordinal, and scale, the latter of which combines interval and ratio) 

Role: a field in which you can assign variables as Input, which means as a predictor or 
independent variable; as Target, which means a dependent variable; as Both; or as None 


ece sa Examole {DataSet} - IBM SPSS Statistics itor 


So eee oy ee 


AAAA A 


Name Type Width Deci... Values Missing Col... Align Measure Role 
1 limited_space Numeric 8 0 More space for more description None None 5 5 é 
2 sex Numeric 8 0 What was your biological sex at birt... {1, Male}... None 5 Æ Right Ordinal @ Target 
3 gender Numeric 8 © With what gender do you identify?  {1, Masculine}... None 8 æ Center & Nominal 
4 sexorientat... Numeric 8 0 What is your sexual orientation? {1, Straight/Heterose... None 13 ægi y Scale itear 
5 race Numeric 8 o What is your race/ethnicity? {1, White/Caucasian}... None 5 MRight Scale split 
6 pol Numeric 8 0 What is your political orientation? {1, Very liberal}... None 5 Right #Scale input 
7 educ Numeric 8 0 What is the highest level of educati... {1, Some high school... None 6 MRight #Scale input 
8 income Numeric 8 0 In which of these groups did your t... {1, UNDER $10,000}... None 7 MRight #Scale input 
9 region Numeric 8 0 What is your region of residence? {1, New England (Mai... None 7 MRight #Scale input 
10 voyesno Numeric 8 0 Have you regularly played video ga... {1, Yes, | regularly pla... None 17 MRight #Scale input 
11 vglname String 2000 0 What is the name of your first favor... None None 15 ELeft  & Nomi... \ input 
12  vgicatname String 80 0 Genre of first favorite game. None None 11 ELeft æ Nomi... \ Input 
13 vglloc Numeric 8 0 Where did you play this game most ...{1, On a video game c... None 8 WRigh #Scale input 
14 vgloften Numeric 8 (J In the last year or two, how often di... {1, 1 ~ Rarely}... None 9 MRighh #Scale input 
15 vglviolent Numeric 8 0 How would you rate the amount of ... {1, 1 - Little to No Vio... None 10 MRigh #Scale ‘Input 
16 vg2name String 2000 0 What is the name of your second fa... None None 15 Bleft æ Nomi... \ Input 
17 vg2catname String 80 0 Genre of Second favorite game. None None 11 Bleft & Nomi... \ Input 
18 vg2loc Numeric 8 0 Where did you play this game most ... {1, On a video game c... None 10 MRigh #Scale ‘Input 
19 va2often Numeric 8 0 In the last year or two, how often di... {1, 1 - Rarely}... None 10 MRight #Scale ‘Input 
20 vg2violent Numeric 8 0 How would you rate the amount of ... {1, 1 ~ Little to No Vio... None 10 MRighh #Scale ‘Input 
21  vg3name String 2000 0 What is the name of your third favo... None None 15 Bleft ANomi... œ Input 
22 vg3catname String 80 0 Genre of third favorite game. None None 1s Bleft  & Nomi... \ Input 
23 va3loc Numeric 8 0 Where did vou plav this ame most ... {1. On a video aame c... None 9 MRiaht #Scale input 
oom vew MSME 
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Figure 9.1.4 SPSS layout for Variable View 


Source: Stephen Warren 
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If you are using the datafile that is provided for you as an ancillary material associated with this 
book, you will notice that we removed the dummy variables and the index variables shown 
here from the datafile. This is so that you can create them yourself when using the datafile to 
practice SPSS commands. 


Dealing with Missing Values 


Sometimes when you collect data, you will have missing values, which is the term used to 
describe gaps in the data set for which there is no meaningful data to report. Missing values 
can occur because research participants fail to provide data for some measures. They may 
leave one or two items blank on a survey, for instance. Or missing values can occur because 
the researcher has made a mistake in data entry or otherwise lost track of some of the data. 

Some advocate for the practice of substituting a missing value for a variable with the mean 
or the median of that variable, a process known as “imputing” the missing value. That discus- 
sion is a little too specialized for this book, and instead, we focus on how to conduct statistical 
analyses with missing values removed rather than replaced. You might be tempted to enter 
zeros for missing data, but this is generally not a good idea, since those zeros will be treated 
as meaningful responses in the statistical analysis even though they are not. And if zeros are 
considered meaningful responses in the data, including them in data analysis will render your 
statistical analyses untrustworthy. 

An additional scenario in which missing data arise is when asking respondents to agree 
or disagree with a statement in survey research using a Likert scale, such as “I think there 
should be more starring roles for Latinx actors in Hollywood films.’ We would typically 
assign 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 
5 = strongly agree as meaningful values. But perhaps we want to give respondents the chance 


2 Example Data sav [DataSet] - IBM SPSS Statistics Data Editor 


‘ae z ESH 36 2, Se 4 ga wt ela 


Name Type Width Deci... Label Values Missing Col... Align Measure Role 

1 age Numeric 8 0 Before we begin, please give your a... None Non = 5 i X Input 
2 sex Numeric 8 (3 What was your biological sex at birth? {1, Male}... None ~X Input 
3 gender Numeric 8 (o) With what gender do you identify? {1, Masculine}... None ~% Input 
4 sexorientat... Numeric 8 0 What is your sexual orientation? {1, Straight/Heterose... None X Input 
5 race Numeric 8 0 What is your race/ethnicity? {1, White/Caucasian}... None X Input 
6 pol Numeric 8 0 What is your political orientation? {1, Very liberal}... None X Input 
7 educ Numeric 8 (o) What is the highest level of educati... {1, Some high school ... None N Input 
8 income Numeric 8 0 In which of these groups did your t... {1, UNDER $10,000}... None N Input 
9 region Numeric 8 0 What is your region of residence? {1, New England (Mai... None N Input 
10 vgyesno Numeric 8 ts) Have you regularly played video ga... {1, Yes, | regularly pla... None X Input 
11 vglname String 2000 0 What is the name of your first favori... None None i... œ Input 
12 vglcatname String 80 0 Genre of first favorite game. None None i... œ Input 
13 vglloc Numeric 8 0 Where did you play this game most ... {1, On a video game c... None 8 MRight #Scale ‘Input 
14 vgloften Numeric 8 0 In the last year or two, how often di... {1, 1 - Rarely}... None 9 MRight #Scale ‘Input 
15 vglviolent Numeric 8 0 How would you rate the amount of ... {1, 1 - Little to No Vio... None 10 MRight Z Scale input 
16  vg2name String 2000 0 What is the name of your second fa... None None 15 E Left Nomi... \ Input 
17 vg2catname String 80 0 Genre of Second favorite game. None None 11 Left Nomi... \ Input 
18 vg2loc Numeric 8 0 Where did you play this game most ... {1, On a video game c... None 10 MRight #Scale input 
19 vo2often Numeric 8 0 In the last year or two, how often di... {1, 1 - Rarely}... None 10 @MRight #Scale input 
20 va2violent Numeric 8 0 How would you rate the amount of ... {1, 1 — Little to No Vio... None 10 MRight #Scale input 
21 vg3name String 2000 0 What is the name of your third favo... None None 15 Bleft BNomi... \ Input 
22 vg3catname String 80 0 Genre of third favorite game. None None 15 left Nomi... \ Input 
23 va3loc Numeric 8 0 Where did vou play this ame most... {1. On a video aame c... None 9 Right Z Scale input 
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Figure 9.1.5a Missing values in SPSS 


Source: Stephen Warren 
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to say “I don’t know” as a response option. If we assign all “I don’t know” responses a 6, fol- 
lowing sequentially after the 5, it will seem like they are indicating agreement, since the level 
of agreement rises as values change from 1 to 5. So, assigning “I don’t know” a label of 6 is 
problematic. Similarly, as we have discussed, assigning “I don’t know” a 0 might also cause 
confusion if it is interpreted as no agreement with the statement at all. The convention in 
survey research is to give “I don’t knows” and any other such non-meaningful data a score that 
is well out of the range of the 1 to 5 Likert scale, like an 88 or a 99. That way these responses 
are less likely to be confused with a more meaningful response on the Likert scale. 

Even when assigning “I don’t knows” or other missing values an 88 or a 99 value, however, 
there is an important extra step involved, as well. As you can see in the series of SPSS screen- 
shots in Figures 9.1.5a, b, and c, under Variable View you must indicate missing values by 
telling SPSS which values for the variable in question should be treated as such. If you do not 
identify missing values using this SPSS function, when you run statistical analyses later using 
the variable in question, SPSS will treat the 88 or the 99 as a meaningful value and this will 
throw off the math in the analysis. Following the steps shown, then, you must indicate that, 
for instance, a value of 99 as we have shown here is a missing value for the variable in question, 
age. You do so by entering 99 as a discrete missing value in the pull-down menu that you see 
in Figure 9.1.5b, at which point the value will show up as missing in Variable View, as you 
can see in the third of this series of screenshots. 


Measures of Central Tendency 


Imagine yourself as the proud creator of a brand new set of quantitative data. The first step 
once you have gathered and input the data is to examine and explore it, variable by variable. 
Examining and exploring each variable will help you to (1) understand your data, (2) look 


. SA Example Data sav (DataSet?) - IBM SPSS Statistics Data Editor 


She Dea Akik h MS wt ol 


Name Type Width Deci... Label Values Missing Col... Align Measure Role 
1 age Numeric 8 0 Before Larrea ia alc ing Nome «5 MRight #Scale input 
2 sex Numeric 8 0 What wa None 5 MRight #Scale ‘Input 
3 gender Numeric 8 0 With wh No missing values None 8 MRight #Scale ~ input 
4 sexorientat... Numeric 8 0 What is © Discrete missing values + None 13 Right #Scale input 
5 race Numeric 8 0 What is None 5 @MRight #Scale input 
6 pol Numeric 8 0 -99 L None 5 MRight Scale input 
7 educ Numeric & 0 Range plus one optional discrete missing value :: None 6 MRight Scale © Input 
8 income Numeric 8 0 ie High None 7 Right #Scale input 
9 region Numeric 8 0 i None 7 MRight #Scale input 
10 vgyesno Numeric 8 Discrete value m None 17 Right #Scale input 
11 vglname String 2000 is None 15 Elet BANomi... \ Input 
12 vgleatname String 80 0 nreo ? Cancel (RG) None 11 Left. «Nomi... \ Input 
13 vglloc Numeric 8 0 Where d » None 8 W Right Æ Scale input 
14 vgloften Numeric 8 0 In the last year or two, how often di... i. 1 -~ Rarely}... None 9 MRight Z Scale input 
15 vglviolent Numeric 8 0 How would you rate the amount of ... {1, 1 - Little to No Vio... None 10 MRight #Scale input 
16 vg2name String 2000 0 What is the name of your second fa... None None 15 left Nomi... \ input 
17 vg2catname String 80 0 Genre of Second favorite game. None None 11 Bleft MNomi... \ input 
18 va2loc Numeric 8 0 Where did you play this game most ... {1, On a video game c... None 10 MRight #Scale input 
19 vo2often Numeric 8 £] In the last year or two, how often di... {1, 1 - Rarely}... None 10 MRight #Scale input 
20  vgviolent Numeric 8 (0 How would you rate the amount of ... {1, 1 - Little to No Vio... None 10 Righ #Scale input 
21 vg3name String 2000 0 What is the name of your third favo... None None 15 Bleft BNomi... \ Input 
22 vg3catname String 80 0 Genre of third favorite game. None None 15 E Left Nomi... œ Input 
23  va3loc Numeric 8 o Where did vou play this aame most... {1, On a video aame c... None 9 WRiaht Z Scale ~ input 
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Figure 9.1.5b Defining missing values 


Source: Stephen Warren 
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2 Example Data sav [DataSet] - IBM SPSS Statistics Data Editor 
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Name Type Width Deci... Label Values Missing Col... Align Measure Role 
1 age Numeric 8 0 Before we begin, please give your a... None Scale = Ss Input 
2 sex Numeric 8 (o What was your biological sex at birth? {1, Male}... 2 Scale ~ Input 
3 gender Numeric 8 0 With what gender do you identify? {1, Masculine}... i # Scale S Input 
4 sexorientat... Numeric 8 0 What is your sexual orientation? {1, Straight/Heterose... 2 Scale =~ Input 
5 race Numeric 8 0 What is your race/ethnicity? {1, White/Caucasian)... Scale ‘Input 
6 pol Numeric 8 0 What is your political orientation? {1, Very liberal)... 5 Scale input 
(2 educ Numeric 8 0 What is the highest level of educati... {1, Some high school ... 6 Scale =~ Input 
8 income Numeric 8 0 In which of these groups did your t... {1, UNDER $10,000}... 7 # Scale =~ Input 
9 region Numeric 8 0 What is your region of residence? {1, New England (Mai... None 7 MRight Scale input 
10 vgyesno Numeric 8 0 Have you regularly played video ga... {1, Yes, | regularly pla... None 17 MRight Scale input 
11 vglname String 2000 0 What is the name of your first favori... None None 15 Bleft  & Nomi... \ Input 
12 vglcatname String 80 0 Genre of first favorite game. None None 11 Left Nomi... \ Input 
13 vg lloc Numeric 8 0 Where did you play this game most ... {1, On a video game c... None 8 Right #Scale input 
14 vgloften Numeric 8 0 In the last year or two, how often di... {1, 1 - Rarely}... None 9 MRight #Scale input 
15 vglviolent Numeric 8 (o) How would you rate the amount of ... {1, 1 - Little to No Vio... None 10 MRight Scale input 
16 vg2name String 2000 0 What is the name of your second fa... None None 15 left Nomi... \ Input 
17 vg2catname String 80 (:] Genre of Second favorite game. None None 11 E Left Nomi... > input 
18 vg2loc Numeric 8 0 Where did you play this game most ... {1, On a video game c... None 10 MRight Scale input 
19 vgĉoften Numeric 8 0 In the last year or two, how often di... {1, 1 - Rarely}... None 10 MRight Scale input 
20 vo2violent Numeric 8 0 How would you rate the amount of ... {1, 1 - Little to No Vio... None 10 MRight Scale input 
21 vg3name String 2000 0 What is the name of your third favo... None None 15 Bleft  & Nomi... \ Input 
22 vg3catname String 80 (o Genre of third favorite game. None None 15 E Left Nomi... \ Input 
23  va3loc Numeric 8 0 Where did vou play this ame most... {1. On a video aame c... None 9 WRiaht #Scale input 
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Figure 9.1.5c Missing values displayed 


Source: Stephen Warren 


for any problems or issues in the data, and (3) determine whether your data array in certain 
patterns that are necessary for further statistical tests. 

One important way to explore your data is to examine measures of central tendency, 
which are ways to express the most common values within a set of values of a variable. There are 
three commonly used measures of central tendency. The mode is the most frequently occurring 
value. If you order the values on any given variable from lowest to highest, the median is the 
value that appears in the middle, with half of the values higher and half lower. The mean is the 
mathematical average of the values, calculated by adding the individual values together and then 
dividing by the number of values. The mean is often abbreviated with a capital M. 

Let’s look at an example. The following text shows ten cases from a survey study of ado- 
lescents in which they were asked to report their age. The mode would be the age reported 
most frequently across the ten cases: 16. The median would be the middle value when the 
cases are ordered from youngest to oldest, 13, 13, 14, 14, 15, 15, 16, 16, 16, 17: 15. The mean 
is calculated by adding the values and dividing by ten: 14.9. The three different measures of 
central tendency in this example give us three slightly different answers to the same question: 
What was the most typical age in our sample? 


Age 
13 
14 
13 
17 
16 
16 
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7 16 
8 14 
9 15 
10 15 


The mean is usually the gold standard in quantitative research because it is a more precise 
measure than the mode or the median. It uses each individual value to summarize across all of 
the values. Most often, researchers will report the mean rather than the mode or the median 
when describing each of the variables in their research. However, there are times when the 
mean can be misleading. When the data have extreme scores—called outliers—those scores 
can skew the mean. When there are outliers in the data throwing off the ability of the mean 
to summarize the typical value, it is best to use the median instead of the mean when report- 
ing central tendency. 

In our example of ten ages reported by survey respondents, for instance, imagine that rather 
than reporting being 15, respondent #10 reported being 35 (clearly not an adolescent!). The 
mode is still 16, and the median is still 15, but now, with the outlier present in the data, the 
mean is 16.9. If we summarized the age variable using the mean and reported an average age 
of 16.9 or even rounded up to 17, that would be true but also a bit misleading since the pres- 
ence of that 35-year-old is skewing the mathematical average upward. Once again, then, to 
correct this, we would usually report the median rather than the mean and state that there was 
an outlier. Alternatively, some researchers would advocate for throwing out the case of the 
outlier and sticking with the 9 respondents whose ages are more similar. In our hypothetical 
example, if we wanted to study adolescents, we should, indeed, throw out the case of the 
35-year-old. In order to “throw out the case,’ we could tell SPSS that 35 should be consid- 
ered a missing value for age, so that all subsequent analyses can be performed with that one 
35-year-old removed. 


> 
gg 
G) 


13 
14 
13 
17 
16 
16 
16 
14 
15 
10 35 an outlier 
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Throwing out an outlier case on a variable such as age may seem fairly straightforward and 
non-controversial. But what if the outlier were the only trans person or the only indigenous 
person in the sample, for instance? We would be remiss if we failed to point out in a book 
about quantitative methods and social justice that identifying someone as an outlier in the 
context of a study and discarding their data due to that identification is a decision that has 
important implications. As we have established in prior chapters, if we exclude data from 
marginalized and/or minoritized populations, we silence their voices at the same time that 
we reinforce the voices of those in dominant or majority populations. You should think 
very carefully about whether to discard outlier cases in your research, then. And just as we 
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established in prior chapters, ideally you would be conscious when sampling to include more 
than a single person from a social group. 

When the mean, median, and mode are all the same, if you plot the values in the data, 
you will see that they form a perfectly symmetrical normal distribution. If they vary, we say 
that the data are skewed. When the mean is greater than the mode, the data are positively 
skewed. When the mode is greater than the mean, the data are negatively skewed. SPSS 
has many ways to use charts, tables, and other functions to explore your data. We will discuss 
one popular choice, a histogram, using Figure 9.2.1. 

Now let’s use the variables in the sample data set to see how SPSS can be used to calculate 
these measures of central tendency. Figures 9.2.2a through 9.2.2e show, step by step, how to 
obtain mode, median, and mean from the SPSS pull-down menus as well as how to check 
to see whether the data are skewed. Figures 9.2.2f and 9.2.3 show the output associated with 
these tests for some of the variables from the sample data set. 

Selecting Descriptive statistics —> Frequencies, as is apparent in the first two visuals in this 
series, will bring up the entire list of variables in the data set. From among that list, you can 
select those for which you want SPSS to calculate mode, median, and mean by selecting or 
dragging some variables from the long list on the left into the box at the right, as seen in 
the third visual in this series. The fourth visual shows the dialog box that comes up when 
you select Statistics, and you can see here that we have asked SPSS to report Mean, Median, 
Mode, Standard Deviation, Minimum value, Maximum value, the Range of values and Skew- 
ness by checking off those boxes among the other options. The fifth visual shows how to 
select the Chart function and ask for histograms, which are handy ways of looking visually at 
whether variables are skewed. The Histogram function uses bars to represent values in the data 
for the variable in question, with different heights of the bars indicating the frequency with 
which that value appears in the data. As you can see in the fifth visual, you can also request 
that a normal curve be superimposed on the histogram so that you can see how the data array 
around the central points. 

The second-to-last visual in this series shows the numerical results for the Mean, Median, 
Mode, Standard Deviation, Minimum value, Maximum value, the Range of values, and 
Skewness for each of the three variables selected from the sample data set to demonstrate these 
functions. And the final visual displays the histograms requested, with one variable showing a 
negative skew, one a positive skew, and one no skew at all (a normal distribution). 


Mean 
Median Median Median 
Mode 
Mode 1- Mean i Mean ~-ı Mode 
i i 
I U 
Positive Symmetrical Negative 
Skew Distribution Skew 


Figure 9.2.1 Skewed and normal distributions 


Source: Diva Jain, https://codeburst.io/2-important-statistics-terms-you-need-to-know-in-data-science-skewness-and- 
kurtosis-388fef94eeaa 
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Reports > 
Descriptive Statistics > ™® Frequencies... 
Bayesian Statistics > E5 Descriptives... 
Tables > “+ Explore... 
Name Type Width Deci... Compare Means > * Crosstabs. 
1 age Numeric 8 0 General Linear Model > Æ TURF Analysis 
2 can Numericl8 o Ganeasa Linear Models > Bw Ratio... 
~ ixed Models > P-P Plots. 
3 gender Numeric 8 0 Correlate > = Q-Q Plots. 
4 sexorientat... Numeric 8 0 Regression > 4 y 
5 race Numeric 8 0 Mita  Loglinear > 
6 pol Numeric 8 0 What ER amoa = 
7 educ Numeric 8 0 LEI@ Dimension Reduction Mome high school ... 
8 income Numeric 8 0 Scale LANDER $10,000}... 
9 region Numeric 8 (o Nonparametric Tests ew England (Mai... 
10 vgyesno Numeric 8 0 Foncea 2 S, | regularly pla... 
Survival > 
11 vglname String 2000 0 Multiple Response > 
12 vglcatname String 80 0 ¥ Missing Value Analysis. 
8 vglloc Numeric 8 0 Caner RT Š a video game c... 
vgloften Numeric 8 0 Ih Simulation - Rarely}... 
15 vglviolent Numeric 8 0 Quality Control = Little to No Vio... 
16 vg2name String 2000 0 Spatial and Temporal Modeling. 
17 vg2catname String 80 0 Direct Marketing 
18 vg2loc Numeric 8 0 Where did you play this game most ... {1, On a video game c... 
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Measures of Variability 


Measures of variability determine the degree to which values are spread out or, conversely, 
clustered together. The standard deviation (usually abbreviated SD) is the most important 
measure of variability, calculating how tightly or loosely dispersed a set of values are for each 
variable. It is calculated by first determining the mean, and then, for each number in the data 
set, subtracting the mean and then squaring whatever value results. Once you have a list of 
squared differences of each value from the mean, the mean of those numbers is calculated. 
Finally, the square root is determined, and that is the standard deviation. Now that we have 
explained that, however, you can promptly forget it if you choose. The computer will calcu- 
late the standard deviation for you. The only thing you need to be able to do is to understand 
what it means. 

Means and standard deviations go together like peanut butter and jelly (or cheese and 
crackers, or chips and salsa, or . . . you get the point!). Every time you report a mean in 
your research, you should also report the standard deviation. This is because the mean tells 
you the average, but the standard deviation tells you whether the set of values used to deter- 
mine the average was tightly or loosely spread around the average. This is important, not 
just mathematically but also for the interpretation of the findings. If a standard deviation is 
large, it means there was considerable variability in the variable. That is not a bad thing (or 
a good thing), necessarily. It just means that you should keep that variability in mind when 
you report the findings. In fact, if there is variability present (as indicated by a high standard 
deviation) on a variable of importance in your study, you might want to think about why 
that is the case and perhaps consider following up to find out more about that variation 
with a new study. 

Finally, it is important to recall that for some variables, the mean does not actually make any 
sense. For nominal-level variables, the values are assigned numbers in a non-meaningful, arbi- 
trary manner. Therefore, you cannot calculate a meaningful mean. In these cases, in order to 
provide univariate statistics, you would determine the frequency of each response rather than 
calculate a mean. Here is an example from a fictitious set of ten respondents who were asked 
to provide their gender identity. The numbers chosen to place respondents into shared cat- 
egories based on their answers are arbitrary. It would make no sense to add these numbers up, 
divide by 10, and report an average gender identity for this sample. Rather, we would report 
the gender identities reflected in the sample in terms of the frequencies of each value, saying 
something like this: 4 of the 10 respondents identified as female (40%), 4 identified as male (40%), 
1 identified as a transwoman (10%), and 1 identified as non-binary (10%). These are frequencies 
and, as you see here, they can be shown as both counts (e.g., 4 out of 10) and percentages 
(e.g., 40% of the sample). 


Gender identity Gender identity, expressed as a number 
1 female 1 
2 female 

3 trans woman 
4 female 

5 male 

6 non-binary 
7 male 

8 female 

9 male 

10 male 
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Frequencies are essential when you are summarizing nominal-level measures. But they are also 
useful when you want to examine your data to see if any issues or problems might be appar- 
ent. Let’s say, for instance, that in your survey, you used the commonly employed 1 = strongly 
disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree for- 
mat to measure respondents’ endorsement of some statement. You can run a frequency test 
on that variable to make sure that everyone chose a response that does, indeed, vary from 1 to 
5. If you run a frequency test and you see a 7, for instance, that value is out of the specified 
range and is therefore not able to be meaningfully interpreted. Chances are, you will need to 
discard that case by treating that value as a missing value and carrying on the rest of the statis- 
tical analyses without it. As you have learned, in SPSS, you can designate any value a missing 
value with just a couple of clicks, and all the statistical analyses you run after that point will 
exclude that value. 

Figures 9.2.4a through 9.2.4d feature a series of screenshots from SPSS that show how to 
run a frequency analysis and what the output of that test will look like. First, you can see 
that the command is under Analyze Descriptive Statistics Frequencies. That will open a 
dialog box with all of the variables from the data set on the left, and you can select however 
many variables you want to determine frequencies for and move them to the right (as the 
third visual shows). The last visual in this set shows the results of a frequency analysis for two 


variables, age and agreement with the item, Men should not be interested in talk shows such as 
Oprah. 
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Figure 9.2.4d Output for Frequency tables 


Source: Stephen Warren 


You can see that the output gives you the frequency of each value for each of these vari- 
ables as they occurred in the data, followed by three columns of percentages. The first is the 
simple percentage—calculated from the frequency compared to the sample size. The second 
is called the valid percentage, and in this case, it has taken any missing values into account. 
(In our examples, there are no missing values, so the first two percentage columns are exactly 
the same.) The final column is the cumulative percentage which displays how much of the 
sample has been accounted for as each new value is added to the last. The valid percent is what 
is most often reported, so that is the most important component of the output. 

This would be a good time to practice running means and standard deviations on inter- 
val and ratio level data and frequencies on nominal and ordinal level data in SPSS. Use the 
descriptive statistics pull-down tab to find these functions, and explore, as well, the ways in 
which you can check for skewness or other properties of the data. 


Scholar Spotlight: Dr. Shardé Davis 


Shardé Davis is an assistant professor in the Department of Communication, Faculty 
Affiliate of the Africana Studies Institute and the Institute for Collaboration on Health, 
Intervention, and Policy (InCHIP) at the University of Connecticut. Dr. Davis (PhD, 
University of Iowa) is an interpersonal communication scholar who specializes in racial 
and gender identity, resistance, counter hegemony, resilience, and intra/intergroup 
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Figure 9.3.1 Scholar Spotlight: Dr. Shardé Davis 


dynamics. She studies how identities, social forces, and cultural conditions interact to 
shape the ways in which Black women communicate with others. 

Dr. Davis has published in Communication Monographs, Human Communication Research, 
Journal of Social and Personal Relationships, Health Communication, Women’s Studies in Com- 
munication, and Review of Communication, among other outlets. She serves as the Chair of 
the African American Communication and Culture Division (AACCD) of the National 
Communication Association as well as for many non-profit organizations that support 
Black women and other people of color in the community. 


Q: You use a wide variety of methodological approaches in your research. How do you decide 
what method to use for any given study? 

Dr. Davis: I let the phenomenon of interest speak for itself. Meaning, I hone my 
research question(s) and think about which method will allow me to uncover 
and make more precise conclusions about Black women’s authentic lived expe- 
riences, with a specific focus on their interpersonal communication practices. 
Establishing a program of research that is phenomenon-driven requires a bit of 
methodological versatility, and fortunately I leveraged my graduate program 
to take classes in a wide range of methods and analytical approaches so that 
I can be nimble during the research process, from study design to the composi- 
tion of manuscripts. For example, the survey method allows me to gather data 
from a large and diverse sample of Black women and determine the extent 
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to which certain communication practices associate with identity-related phe- 
nomena (e.g., how Black women’s regulation of strength at the group-level 
affects the conveyance of social support). Controlled experiments, on the other 
hand, allow me to identify causal relationships between Black women’s unique 
communication style and important outcomes, such as psychological function- 
ing, physiological responses, and self-esteem. Also, content analyses allow me 
to systematically code for the frequency of certain words or behaviors during 
an interaction to understand Black women’s communication practices during 
micro-events. I can also content analyze the transcribed interaction and use 
language software (i.e., Linguistic Inquiry Word Count) to calculate the degree 
to which women use positive or negative emotions, self-references, and other 
linguistic dimensions. 

Qualitative methods are also utilized in my work because these techniques 
privilege the voices of my informants. With this approach, I can inductively 
explore the communication narratives that are not fully understood or articu- 
lated by interpersonal scholars. While numerous social researchers employ quali- 
tative methods, not all qualitative work is oriented toward social justice. In some 
of my research, however, I use qualitative methods that subscribe to the feminist 
method of inquiry. This primarily includes Black women participants narrat- 
ing their own stories (instead of me making blind, objective observations) and 
infusing my values as a resource to the research. My primary qualitative methods 
are focus groups, in-depth quasi-structured interviews, and participant observa- 
tion/video recall. Focus groups promote uninhibited interactions among Black 
women, which create an ideal environment to explore Black women’s atti- 
tudes and beliefs. Participant observation and video recall are ideal methods for 
my work on collective communication practices because the women are dem- 
onstrating their communication behaviors in real time and in a more natural 
manner. Complementing observation with video recall allows the women to 
interpret the interaction from their viewpoint and ascribe meaning to their own 
behaviors. 

I recognize that many scholars prioritize depth over breadth when it comes 
to learning and attuning their methodological skills. I have taken a different 
approach in my work. Black women’s communication is inherently complex 
because their behaviors are a function of their racial-gendered identity, the 
macro- and micro-level discrimination they encounter because of these (and 
other) intersecting identities, and their marginal position in dominant soci- 
ety. The research needs to be approached with great creativity and innovation 
so that their communication behaviors are appropriately contextualized and 
interpreted. 

Q: Do you think statistics or statistical analysis have a particular contribution to make to social 
justice efforts? Why? 

Dr. Davis: Contrary to popular belief, quantitative methods and statistical analyses 
do not have to be the antithesis of social justice and feminist efforts, which col- 
lectively aim to destabilize power hierarchies and redistribute power and other 
resources in a more equitable fashion. Some feminist researchers even go as far as 
to recognize the merit of scientific methodology, specifically its predictive accu- 
racy, testability, and technical analysis. They believe it “provides [scholars] with 
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valuable purchase in the dominant culture and access to the power of scientific 
resources” (Hundleby, 2012, p. 41). In my work, quantitative methods and statisti- 
cal analyses are used strategically to challenge status quo research and identify the 
embeddedness of inequality within communication measures, statistical models, 
and theory. Other scholars use the strengths of science as a departure point to 
theorize about knowledge in ways that do not make gender, racial, or other biases 
and to “provide credible numbers that can help advocates mobilize political sup- 
port” (Spalter-Roth & Hartmann, 1996, p. 221). All things considered, embrac- 
ing the scientific tradition in communication research can provide the cachet to 
heighten consciousness about pertinent social issues and effect necessary change, 
making it a viable avenue for any justice-oriented, feminist, and/or critically- 
engaged scholar. 

Q: In your view, what are the limits to using statistical analysis for social justice? What should 
scholars using statistics try to do/not do or try to keep in mind in order to be critically conscious? 

Dr. Davis: Indeed, there are some glaring challenges in using quantitative methods 
and statistical analysis in social justice research. But any scholar can maneuver these 
challenges by being thoughtful and intentional about each decision made during 
the research process. I situate my comments under the work of prior feminist social 
scientists. 

Social science research from a (post)positivistic perspective subscribes to the 
logic of scientific inquiry and the tenets of objectivity, verification, and generaliz- 
ability. Relatedly, (post) positivists believe that they can make neutral, universal 
scientific claims about the social world when they follow certain methodological 
protocol, such as separating the researcher’s feelings and emotions from the sub- 
ject. Abiding by this protocol allows science to be logical and rational. Also, (post) 
positivist research is typically conducted in male/White-dominated social spaces 
and, consequently, focuses on White/male-centered problems. 

Feminist scholars reject the notion that the researcher should be removed from 
the data and have posed long-standing critiques that (post)positivistic research, 
specifically its commitment toward an objective study of the natural world by 
separating empirical knowledge from the politics of communities, excludes 
diverse sources of experience (Hesse-Biber, 2012; Hundleby, 2012). Instead, fem- 
inist scholars advocate for a more reflexive and relational approach to knowledge 
building, whereby researchers negotiate their positionality, engage informants in 
conversation about their personal experiences, and address the power imbalances. 
Knowledge is situated and fluid; thus feminist researchers should seek new ways 
to transform how it is conducted. 

Feminists also believe that post-positivist research attends to the concerns and 
social values of dominant social groups (i.e., White Americans, middle/upper 
class, men) and ignores the concerns of minority groups (i.e., communities of 
color, lower/working class, women), thereby stifling the potential for “break 
through conceptualizations” of non-dominant social experiences (McHugh, 
Koeske, & Frieze, 1986, p. 879). If the scientific paradigm is laden with power, 
then the neutrality and universality of scientific research is called into question. 
This discrepancy prompted many feminists to assert that the scientific model can- 
not serve as an objective approach to knowledge building because the claims 
are based on distorted data (Hesse-Biber & Piatelli, 2012). Scholars who wish 
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to pursue research at the margins should recruit people from underrepresented 
groups into research studies because that allows for a variety of voices and research 
issues to be centered and, thus, produces more trustworthy data. 

Q: What advice would you give an emerging scholar who would like to work toward making 
their research matter? 

Dr. Davis: My advice to emerging scholars is to follow your gut and create knowl- 
edge that sits well with your soul. Sometimes doing so will mean that you produce 
work that is challenged or even undermined by your colleagues. But keep going. 
In the early parts of my career, scholars did not understand why my interdisci- 
plinary and critically inflected approach to research on Black women was neces- 
sary. But the scholarly conversation has since shifted, and those same colleagues 
now realize the value of my work and how it enhances the area of interpersonal 
communication and the larger field. I would not have reached this point in my 
career if I listened to the nay-sayers and did work that cohered with the domi- 
nant trend. I encourage any budding scholar to let your gut guide the decisions 
you make because I believe it will never steer you in the wrong direction. With 
that said, pursuing research that is socially meaningful and transformative may 
require methodological versatility and theoretical diversity. Part of the reason is 
because (post)positivism has been considered the dominant paradigm in the social 
sciences, and not surprisingly its research and theories typically constitute the 
“mainstream.” A bias toward this kind of work means that standards for “good 
science” privilege (post)positivist research designs, methods, samples (including 
sample size), and analyses, which can stymie work “at the margins.” While (post) 
positivism has some merit and value, scholars should not be afraid to exercise 
agency and deviate from certain empirical standards when other bodies of work 
call for a pivot (i.e., critical cultural studies, race studies, feminist studies, research 
on social justice, and grassroots activism). Embracing methodological creativity 
and digressing from the theoretical canon extends ripe opportunities for research- 
ers to discover ideas that would not have been realized otherwise. 


Data Reduction and Transformation 


After you have investigated your variables one by one, you may want to transform them or 
combine them in some ways before moving on to bivariate and multivariate analyses. We will 
discuss four very common ways to change your variables to make them more useful in subse- 
quent analyses. For each of these transformations, we will introduce situations in which you 
might consider using these approaches. 


Recode 


First, a very useful SPSS function is recode. Using the recode function, you can change the 
values from a variable into new values, and SPSS will either replace the first variable with 
this new one (using the function called Recode Into Same Variables) or save them both 
(Recode Into Different Variables). Recoding is useful when you want to collapse data from 
many categories into fewer categories. For instance, if you measured political ideology from 
1 = very conservative to 7 = very liberal, as was the case in the sample data set (and as seen in 
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Figure 9.4.1), perhaps there were very few respondents who identified as 1s or 7s. In order to 
have fewer categories with more people in each, then, you might want to change from seven 
values for political ideology to three by collapsing those who had responded with a 1 or a 2 
with those who had responded with a 3, and by collapsing those who had responded with a 
6 or a 7 with those who had responded with a 5. In doing so, we put those who were “very 
liberal” as well as those who were “slightly liberal” together with those who simply said they 
were “liberal; and we would do the same for the “conservative” values, as well. 

This function can be found under the Transform tab, as you can see in Figure 9.4.2. That 
opens up a new dialog box, as you can see in Figure 9.4.3, in which you select the original vari- 
able for political ideology from the entire list of variables using the little arrow, which will then 
make that variable populate the left-hand side of the new dialog box. The next step is to give 
the new variable you are creating through the recoding of political ideology a name and a label. 
In this case, we have chosen to call it pol_recode, Political Ideology Recoded, as you can see in 
Figure 9.4.4. After naming and labeling the new variable you are creating, you click Change. 

Next, you click Old and New Values, as seen in Figure 9.4.5. That brings up the dialog 
box you see in Figure 9.4.6, in which you proceed to tell SPSS how to handle every value 
from the original variable in the formation of the newly recoded variable. For instance, we 
would tell SPSS that a value of 1 for the original variable, Political Ideology, is also a 1 in the 
new variable, Political Ideology Recoded. A value of 2 in the original variable is a 1 in the 
new variable, and a value of 3 in the original is also a 1. The original 4 value (which was the 
midpoint in the original 7-point measure) now becomes a 2 (the midpoint in the new three- 
point measure). The original 5 becomes a 3, as do the original 6 and 7. If there were any 
missing values, identified in the dialog box as System Missing and in the output as SYSMIS, 
you would once again have to tell SPSS how to handle the missing data in the new variable. 
Once this process is accomplished, as seen in Figure 9.4.8, you hit Continue and SPSS creates 
the new variable, Political Ideology Recoded. 


@ Example Data sav [DataSet] - IBM SPSS Statistics Data Editor 
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Name Type Width Decimals Label Values Missing Columns Align Measure Role 
1 age Numeric 8 (o) Before we be... None None 5 W Right 2 Scale X Input 
2 sex Numeric 8 (o What was you... {1, Male}... Non W Right 2 Scale X Input 
3 gender Numeric 8 (eJ With what ge... {1, Masculi.. W Right ? Scale ~X Input 
4 sexorienta... Numeric 8 (0) What is your ... {1, Straigh.. 13 Æ Right 2 Scale X Input 
5 race Numeric 8 0 What is your ... {1, White/... 5 Æ Right # Scale X Input 
6 pol Numeric 8 0 What is your ... 5 Right 2 Scale X Input 
7 educ Numeric 8 = ace re See # Scale X Input 
B income Numeric 8 @ ® Vale Labais, $ Scale N Input 
9 region Numeric & -Value Labels @ Scale X Input 
10  vgyesno Numeric 8 7 # Scale N Input 
11 vglname String 2 Noon Spelling... & Nominal X Input 
12 vglcatname String 8 Label: & Nominal X Input 
13 vglloc Numeric 8 3 # Scale ~X Input 
14 vgloften Numeric 8 1 = ‘Very liberal” 2 Scale ~X input 
15 vglviolent Numeric 8 2 = ‘Liberal # Scale ~X Input 
16 vgłname String 2 Add 3 = "Slightly liberal & Nominal X Input 
17 vg2catname String 8 4 = “Moderate” & Nominal X Input 
18 vg2loc Numeric 8 Change 5 = “Slightly conservative” Ê Scale X Input 
19  vg2often Numeric 8 Roana 6 = "Conservative" A 2 Scale N Input 
20 vgžviolent Numeric & 7 = Very conservative 2 Scale N Input 
21 vg3name String 2 @Nominal 6 Input 
22 vg3catname String 8 & Nominal X Input 
23 va3loc Numeric 8 # Scale X Input 

E | Help Cancel OK s 


i SPSS Statistics Processor is ready —— Unicode:ON 


Figure 9.4.1 The existing values for a selected variable 


Source: Stephen Warren 
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Figure 9.4.2 The Recode into Different Variables command 
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Figure 9.4.3 Select the variable to recode 


Source: Stephen Warren 
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9.4.4 Name and label the new variable being created 
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Figure 9.4.5 Click to insert old and new values 
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Figure 9.4.6 In this example, a value of 1 on the initial variable remains a 1 for the new variable 
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Figure 9.4.7 The old and new values specified will appear in the dialog box 
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Figure 9.4.8 The dialog box now shows all of the old and new values that were created step by step 
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Figure 9.4.9 SPSS is now ready to create the new, recoded variable 
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Figure 9.4.11 The final step in recoding 
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As you can see in Figure 9.4.9, however, you do need to enter labels for the new values of 
the newly created variable and then click OK (Figure 9.4.10 and Figure 9.4.11). It is a good 
idea to run a frequency test on the newly created variable to make sure it has been formed 
properly, as is shown in Figure 9.4.12. Finally, Figure 9.4.13 shows the frequency table for the 
original variable alongside the frequency table for the new variable, and you can see how the 
data were collapsed from the larger number of categories with fewer responses in each to a 
smaller number of categories with more responses. 


Creating an Index and Testing Its Reliability 


As you learned in Chapter 3 on measuring variables, often researchers use a number of items 
to measure the same concept. Doing so improves reliability and validity. Rather than using 
every single one of those items in subsequent statistical analyses, researchers typically want 
to fold them all into one, simple, easy-to-use measure. If the multiple items all measure the 
same concept, then you are justified in combining them into a single measure. For instance, 
there are existing scales to measure Islamophobia, consisting of multiple items all designed to 


@ sPssStatistics File Edit View Data Tra rm Analyze G s 
Reports istics Di y 
Descriptive Statistics i Frequencies... 
Bayesian Statistics Pa Descriptives... 
Tables Explore... ent 
Compare Means Crosstabs... Missing C 
i winnie p n a EET 5 
2 Variable(s) ye 5 
5 | Men should never hold hands or show affection toward another. [M&NI_NS. Ê What is your political orientation? [poll Statistics... - 
3 
| ter shu ply wth cto ures ot ot. DARN. AY Re i 8 
4 Ê Men should waach football games instead of soap operas. [MRNIAF2] le 1: 
T | & Boys should prefer to play with trucks rather than dolis. (MRNI_AF3} Format. è 
@ Aman should prefer watching action movies to reading romantic novels. |, 
` A Men should not wear make-up, cover-up, or bronzer, [MRNI AFS] Style. fe 5 
Boys should not throw baseballs like giris. [MRNI_AF6] 
| | @ Men should not be imerested in talk shows such as “Oprah” [MRNLAF?] toot. le 5 
TU 2 Amman should aways be ready for vex. (MINES!) ie 6 
5 & A man should not ture down sex. [MRNLIS2) 
8 @ Men should always like to have sex. (MRNI_IS3) te 7 
9 7, Ween te one ss Seah mee ned AA Mee ie 7 
| I think a young man should try to be physically tough, even if he’s not big. a 
10 @ Men should get up to investigate if there is a strange noise in the house at + le 17 
= EMOn D E Rea ei rene a TA AR hurt. [MRNI_T4) x 
Men should be the leader in any group, [MRNI_D1} 
11 @ A man should always be the boss. [MRNI_D2] le 1} 
12 2 Men should make the final decision involving money. (MRNI_D3} le 1] 
T & The President of the U.S. should always be a man. (MRNILO4) 3 
13 | A man shouid provide the discipline in the family. [MRNI_OS] le 8 
T & Aman should always be the major provider in his family. [MRNI_DS] $ 
14 Ê ina group, it is up to the men to get things organized and moving ahead. re 9 
15 @ if anther man firts with a man's gelfriend, he shouid tell him eff, [ADMI AI) 3 
2 A Semationes, a man’s gor to fight or people wil walk al over him [ADMI A2] ie 1¢ 
Ê ii a man made a pass at another man’s wife, the husband is justified if he 
16 @ W's okay for a man to use physical violence to defend what he has. (ADM te 1s 
17 @ "a man is threatened, he should be ready to fight. [ADMI AS] le 1 
A Otte moane cbt re 1 
19 ? Reset Paste axt DD te + 
20 g2violer umeric 8 0 OW d yor unt of ... ' Little t 0 lone 1¢ 
21 vg3name String 2000 0 What is the name of your third favo... None None 1 
22 3catname Strin 80 0 Genre of third favorite game. None None 1 
vg 
23 vq3loc Numeric 8 0 Where did vou play this game most ... {1, On a video qame c... None 9 
Dataview MAASE 


IBM SPSS Statistics Processor 


Figure 9.4.12 Select the initial variable and the newly created recoded variable to run frequency tests 
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Frequency Table 


What is your political orientation? 


Cumulative 
Frequency Percent Valid Percent Percent 


Valid _Very liberal l | 
Liberal 16.0 16.0 23.7 


Slightly liberal ul : i 9.2 | 9.2 | 32.9 
Moderate 193 46.7 46.7 79.7 

_ Slightly conservative - 30 7.3 7.3 86.9 
Conservative 37 9.0 9.0 95.9 
Very conservative 17 4.1 4.1 100.0 
Total 413 100.0 100.0 


Recoded Political Ideology 


Cumulative 
Frequency Percent Valid Percent Percent 
Valid Liberal l ' 
Moderate 193 46.7 46.7 79.7 
Conservative 20.3 20.3 100.0 
Total 413 100.0 100.0 


Figure 9.4.13 Frequencies for the initial variable and the recoded variable 


Source: Stephen Warren 


tap into the phenomenon. In order to fold them all together to come up with a composite 
measure for Islamophobia, we must first test whether such a composite measure is reliable. A 
composite measure is what we call any measure that is made up of multiple measures. An 
index (a term often used interchangeably with scale) is a commonly used composite measure. 

SPSS has a function called Reliability Analysis in which you can select all of the items 
that you used to measure a single concept (like Islamophobia). When you run a reliability 
analysis, the SPSS output will give you a score for Cronbach’s alpha (or Cronbach’s œ). 
Cronbach’s alpha ranges from 0 to 1.0, and the higher the value, the more reliably the items 
comprising the index hang together to capture the concept. Generally, you will want the 
Cronbach’s alpha coefficient to be .65 or higher in order for the index to be considered reli- 
able. If you get a result that is lower than that, rather than combine the items into a composite 
measure, you should perform your statistical analyses using the individual items themselves. It 
will be less parsimonious because there are multiple tests to run using each of the individual 
items rather than just one composite variable, but it is necessary if the multiple items do not 
hang together reliably. 
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The next set of screenshot visuals from SPSS shows how to test whether multiple items 
cohere reliably together. This is an important first step before actually combining the multiple 
items into an index. First, as you can see in Figure 9.5.1, you select Analyze— Scale Reli- 
ability analysis. In the example we are providing from the sample data set, we are selecting 
the seven items that are designed to measure the Avoidance of Femininity component of the 
Masculine Roles and Norms Index-Revised, as you can see in Figures 9.5.2 and 9.5.3. In Fig- 
ure 9.5.4, you can see the dialog box that this function produces and how to bring the seven 
selected items over to the top right. Next, you click the Statistics button and then ask for 
Descriptives for Item, Scale, and Scale if Item Removed, and hit Continue, as is apparent in 
Figures 9.5.5 and 9.5.6. Figures 9.5.7 and 9.5.8 show the output that SPSS creates when you 
enter these commands. The Item Statistics box shows the means and standard deviations for 
all of the seven items selected as well as the sample size (number of respondents who provided 
valid data for each item). The most important component of the output is the Reliability box, 
and the most important part of that box is the part we have highlighted. Once again, the reli- 
ability statistic is Cronbach’s alpha, and the closer that number is to 1.0, the more reliably the 
items in the analysis cohere around a single concept. In our case, then, the Cronbach’s alpha is 
.929, which would round to .93, which easily surpasses the .65 threshold considered sufficient 
for reliability. This is good news, then, for how well the seven items in question measure the 
Avoidance of Femininity concept, and it means that in subsequent analyses, rather than use 
seven separate variables, we can combine these seven into a composite measure, an index, and 
simply use that single index to represent all seven items. 

There is one last detail to report from the output, and that is indicated by the box around 
the “Cronbach’s alpha if item deleted” portion. This column of numbers shows us whether 
dropping one of the seven items would increase the reliability of the items that remain. In 
other words, perhaps six of the seven items do a good job of measuring the Avoidance of 
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Figure 9.5.1 The variables comprising the Avoidance of Femininity component of the scale 


Source: Stephen Warren 


236 Statistical Analysis 


@ sPssStatistics File Edit View 3 sform Analyze 


ence Reports istics Data Editor 
[Sue Qa £4.47 ees 


Tables 


Name Type Width Deci Compare Means Values Missing Cc 


Classify 
Dimension Reduction 
Scale 
Nonparametric Tests 
Forecasting 
Survival 
Multiple Response 
Numeric ® Missing Value Analysis. icks rather... {1, Strongl 
AF4 [Numeric 5 Multiple Imputation 
Complex Samples 
AF5 Numeric Simulation. 
_AF6 [Numeric Quality Control ke girls. {1, Strongl.. 
MRNI_AF7 Numeric Spatial and Temporal Modeling. alk shows s. , Strong! 


83 MRNI_LNSM6 Numeric None 
84 MRNI_LNSM7 Numeric 


85 MRNI_LNSM8 Numeric 


RI Reliability Analysis... 

8P Multidimensional Unfolding (PREFSCAL)... 
E Multidimensional Scaling (PROXSCAL) 

E Multidimensional Scaling (ALSCAL) 


77 MRNI_SR3 Numeric 8 0 General Linear Model his car if i... {1, Strongl... None 10) 
78 (MRNILNSM1 Numeric 8 0 ahaa oe È {1, Strongl... None 11) 
79 MRNI_NSM2 Numeric 8 0 Correlate ic. {1, Strong]... None 11) 
80 MRNI_NSM3 Numeric 8 0 Regression {1, Strongl... None 16 
81 MRNI_NSM4 Numeric 8 0 Logiinesr e teachin... {1, Strongl... None 12) 
82 MRNILNSMS Numeric 8 0 Arcee ial serve in t... {1, Strongl... None 11 

8 0 

8 0 

8 0 


vvvVYYVVYYvrvevvr 


ion movies , Strong! 
wver-up, or... H1, Strongl 


MRNILIS1 Numeric 8 0 piect Maaing 3 Strongl... 
94 MRNI_IS2 Numeric 8 0 A man should not turn down sex. {1, Strongl... None 
95 MRNI_IS3 Numeric 8 0 Men should always like to have sex. {1, Strongl... None 
96 MRNI_T1 Numeric 8 0 When the going gets tough, men should get ... {1, Strongl... None 
97 MRNI_T2 Numeric 8 0 I think a young man should try to be physica... {1, Strongl... None 
98 MRNI_T3 Numeric 8 0 Men should get up to investigate if there is a... {1, Strongl... None 
99 MRNI T4 Numeric 8 0 It is important for a man to take risks, even i... {1, Stronal... None 


oat view ae 


IBM SPSS Statistics Processor is ready 


Figure 9.5.2 The Reliability command in SPSS 
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Figure 9.5.3 Select variables from the left 
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Figure 9.5.4 The selected variables and the Statistics command 
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Figure 9.5.5 Options to choose in the test for the reliability of a scale 
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Figure 9.5.6 Output for the Reliability test 
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Femininity concept, but one does not. If that were the case, then, dropping that one would 
increase the Cronbach’s alpha score. In our example, however, as you can see, that is not the 
case. None of the seven items should be dropped because doing so would only decrease the 
Cronbach’s alpha, and, therefore, lower the reliability of the subsequent index rather than 
increase 1t. 


Compute 


Although we have organized this next function under a new subheading, it actually connects 
directly back to the earlier discussion about forming an index (and so we’ve numbered the 
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Item Statistics 
Std. 
Mean Deviation N 
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action figures not dolls. 
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Figure 9.5.7 Output for the Reliability test, descriptive statistics for the items 


Source: Stephen Warren 


screenshots as an extension of the last set). There is one last step in the reliability analysis that 
introduces a third important way to transform your variables. It is important to note that the 
reliability analysis function of SPSS simply tells you whether you can or should form an index; 
it does not actually create the index. If you obtain a sufficiently high Cronbach’s alpha on 
selected items that measure the same concept, then, you must then take another step, and use 
the Compute function to actually combine the seven items into a new variable. 

The next five visuals in this series show how to use the compute function to combine the 
seven items that measure Avoidance of Femininity into a single item that is calculated from 
the mean scores of the seven. First, you find Compute Variable under the Transform tab, as 
you can see in Figure 9.5.2.1. As you can see, that opens up a dialog box that looks like a 
calculator and, indeed, is used just the same to choose from among many basic mathematical 
functions. Next, you can right click on the list of variables, which will call up a dialog box in 
which you can change what is displayed from the variable names to the labels, as is shown in 
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Figure 9.5.8 Output for the Reliability test, alpha if item deleted 
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Figures 9.5.2.2 and 9.5.2.3. The labels are generally easier to read, which is why we are show- 
ing how to right click to see the labels here. From the dialog box on the lower right you can 
select the mathematical function you would like to perform. In our example, we are taking 
the mean of all seven items that comprise the Avoidance of Femininity scale. Figure 9.5.2.5 
shows that you must name the new variable that will be created from this function (we have 
gone with MRNI_AF for Masculine Roles and Norms Index Avoidance of Femininity) and 
ensure that all seven variables of interest that you had selected are present in the upper right 
portion of the dialog box. This portion also shows the mathematical function selected (in this 
case, calculating the mean). 

Indeed, Compute is quite a versatile SPSS function. It can create new variables through 
any mathematical process such as adding variables together, multiplying two or more vari- 
ables, taking the average of variables, etc. In addition to making a composite measure from 
the average across each of the items that comprise it as we have shown in our example, other 
common uses include multiplying items to arrive at time estimates (like number of minutes 
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Figure 9.5.2.3 Changing display to variable names 


Source: Stephen Warren 


@ SPSS Statistics 


ese — — - 
Sho Nea w 

Pz Type | Wi : i Values M 
eco %3 
‘Target Variable: 


Compute Vari: 
 Programmab! 
F? Count Values 
alues. 


/ Transformation. 


in Cases 


Type & Label. 


Ë MENI NMS 

Ê MRN NS 

Ê MENIN? 

2 urni Ns 

Ê NANLAFL 

P MENAR 

P MRNAS 

P MENINA 

Ê NENAS 

P MRNAS 1 

P MENNT = 

Ê RNS + 
Ê MRNLIS2 
P MENISI 
Ê nT 
P NT 
2 NTI 
Ê NENTA 


À smn ns 


MEAN(numexpr.numexpr(,..J). Numeric. Returns the arithmetic mean of 
its arguments that have valid, nonmissing values. This function 
requires two or more arguments, which must be numeric. You can 
specify a minimum number of valid arguments for this function to be 
evaluated. 


if... (optional case selection condition) 


99 


ox view Mais 


IBM SPSS Statistics Processo! 


Figure 9.5.2.4 The dialog box with mathematical functions for Compute 
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Figure 9.5.2.5 Computing the mean of multiple items 
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per day spent on Instagram multiplied by number of days a week using Instagram to get at 
total Instagram use) and adding multiple items together to make an additive index. 


Select Cases 


Under the Data tab in SPSS, you will find the Select Cases function. You can use this 
function to temporarily isolate certain cases in your data set to look within those cases. For 
example, perhaps you want to just look at those who identified as politically moderate on the 
political ideology variable for some of your analyses. You can use the Select Cases function to 
choose only those who selected a 4 on the scale of 1 (very liberal) to 7 (very conservative) for 
that variable, and SPSS will temporarily set all others aside. While those who selected 4 are 
active in the data set and all others are not, you can perform any statistical analyses you would 
like within the active group. Once you are done with these analyses, you can go back and 
select all cases again to look across those with various political ideologies once again. 

As you can see in the series of SPSS screenshot visuals that follow, first you choose Data> 
Select Cases from the pull down menu (Figure 9.6.1). This will call up a dialog box in which 
you can choose Select if condition is satisfied (Figure 9.6.2). You select the variable of inter- 
est from the list of all variables in the data set on the left (Figure 9.6.3), and then use the 
keypad that looks like a calculator to choose what condition you would like to select. In our 
case, we chose political ideology = 4, selecting those who identified as politically moderate 
(Figure 9.6.4). Keying in this function will populate the dialog box with this information 
(Figure 9.6.5). Figure 9.6.6 shows what the data file looks like with the Select function active. 
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Figure 9.6.1 The Select Cases command in SPSS 


Source: Stephen Warren 
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Figure 9.6.2 Select if condition is satisfied 


Source: Stephen Warren 
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Figure 9.6.3 Identify a variable to use to determine which cases will be selected 
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Figure 9.6.4 Choose the value that will be used to identify the selected cases 
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Figure 9.6.6 Only cases matching the selection will remain active in the data file 
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The cases of those who did not answer the political ideology item with a 4 are crossed out, 
signifying that they are no longer active in the data set. Those who did answer 4 remain. At 
this point, then, you could proceed with whatever analyses you wanted to perform on only 
those who answered 4. Of course, selecting only those who scored 4 will severely decrease 
your sample size, so you may not be able to run advanced statistical tests that require large 
sample sizes. In any case, once you have finished with those analyses, you would go back to 
the Select Cases function and click on “All” cases. This will reset the data file so that all cases 
are active once again. 


Research in Action 


Statistical Test Decision-Making Guide 
You want to know if two variables are related 
—and they are both nominal level 
— then the test to conduct is cross-tabulation with chi square. 
—and they are both ordinal level 
— then the test to conduct is Spearman’s rho. 
—and they are both interval or ratio level 
— then the test to conduct is Pearson’s correlation. 
—and you want to add a control variable 
= then the test to conduct is a partial correlation. 


You want to know whether multiple variables are related to a single outcome variable 

—and the outcome variable is interval or ratio 

— then the test to conduct is multiple regression. 
You want to know whether two groups differ 

—and the dependent measure is interval/ratio 

— then the test to conduct is a t-test. 

—and the two groups are independent of each other 

— then the test to conduct is an independent t-test. 

—and the two groups are closely related or are the same people tested 
twice 

— then the test to conduct is a dependent (also known 
as a) paired t-test. 
You want to know whether three or more groups differ 

—and the dependent measure is interval/ratio 

— then the test to conduct is a one-way analysis of variance. 
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Recall from Chapter 8 that when you look for patterns with two (bivariate) or more than two 
(multivariate) variables within your sample, you will most likely also be attempting to discover 
whether the results you find in the sample are likely given the null hypothesis. Therefore, 
when you run bivariate and multivariate statistics, you are also running inferential statistics, 
coming up with both an answer to the statistical test itself and an answer for whether the result 
is statistically significant. This would be an excellent time to refresh your memory about what 
statistical significance means by returning to Chapter 8. 

There are many different bivariate and multivariate statistical tests you can run to test 
hypotheses and explore research questions. Learning how to figure out what test or tests to 
run is perhaps one of the most challenging aspects of learning quantitative research methods. 
Thankfully, there is a shortcut that you can use to distill most statistical tests into two basic 
types: those that you use to test whether groups differ and those that you use to test whether 
variables relate. We will discuss some of the main statistical tests that fall into each of these 
two types. We have also created a guide for choosing the right statistical tests in the Research 
in Action section of this chapter. 

Yet, before you begin learning different statistical tests, you should probably revisit (or 
re-revisit!) Chapter 3, to remember that variables are measured, variously, at the nominal, 
ordinal, interval, and ratio levels. Some tests require variables to be at a particular level of 
measurement, so as you learn statistical tests, you will want to be sure you really understand 
and can apply those different levels of measurement. Some tests, too, require that you identify 
an independent variable and a dependent variable. Recall that an independent variable is 
a predictor variable; it is the variable that the researcher assumes is causing an influence 
on another variable. A dependent variable, also called an outcome variable, is what the 
researcher assumes is being affected or shaped by the independent variable. 

Also before we begin in earnest, we should briefly explain degrees of freedom. Many 
(but not all) statistical tests require the researcher to report degrees of freedom alongside the 
test result. Degrees of freedom are labeled as df. Degrees of freedom are the number of values 
that are free to vary when calculating a statistic. Imagine that you have four numbers and they 
have to add up to 22. If the first three numbers are 12, 3, and 5, then the last one must be 2. 
It cannot vary. Thus, the degree of freedom in the test is 3. Three numbers can vary, one is 
fixed. The same logic applies in some of the tests that you will learn how to run, as you will 
see later in this chapter. It is standard practice to report degrees of freedom alongside many of 
the statistical tests we will cover in this chapter. 

Finally, we should note—briefly!—that some of the statistical tests that we are about to 
introduce (including t-tests, analysis of variance, correlations, and regression tests) require that 
the variables examined in the test should be normally distributed. Normality of data should 
be checked prior to running the tests, then, using the tests for skewness and distribution that 
we have already explained. If the condition of a normal distribution of variables is not met, 
the researcher can either correct for the non-normality (like by getting rid of outliers) or can 
choose what is called a nonparametric test that does not require a normal distribution. 
A discussion of the body of nonparametric tests is a bit outside our scope here, but the inter- 
ested reader can read more online. 


Tests of Differences: T-Tests 


Some hypotheses or research questions call for determining whether two groups differ from 
each other on a variable of interest. If the two groups are not closely related to each other (as 
in, the two groups do not constitute people who are romantic partners, or a parent and child, 
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or some other closely related dyad), for these hypotheses, you can use an independent sam- 
ples t-test, also known as an independent t-test. (Please note that “t” is typically expressed 
with a lowercase “t” for this test). An independent t-test is calculated from the difference on 
some variable of interest between the two groups in addition to the size of the two groups and 
the amount of variation both within each group and between the two groups. 

Sometimes you have two groups and they are either closely related to each other (once 
again, such as partnered couples or dyads of kids and parents) or the two groups in a study are 
actually the same people who have completed the same measure(s) twice. If this is the case, 
you can test for differences on a variable of interest between the two groups using a depend- 
ent samples t-test, also known as a dependent t-test or a paired t-test. A dependent t-test 
is derived from the sum of the differences between groups and the sum of the differences 
squared between the groups in addition to the sample size. 

For both the independent and the dependent t-tests, the grouping variable—the one 
that divides the sample into two groups—is usually a nominal level variable. For instance, for 
an experiment, you will need to have a variable that tells you whether the participant was in 
the control condition or the treatment condition. This variable divides the sample into two 
groups (control and treatment group members) and is measured at the nominal level. Because 
t-tests determine whether means are different in the first group compared to the second 
group, the dependent variable in the test must be measured at the interval or ratio level, so 
that the means make sense. 

We will illustrate an independent t-test visually with screenshots from SPSS because those 
tests are more common than dependent t-tests. First, as you can see in Figure 9.7.1, you select 
Analyze Compare Means— Independent Samples T Test from the pull-down menu. Then 
you choose from the list of all variables in the data set that appears on the left which variable 
will constitute the grouping variable. You select that variable and place it in the bottom right 
of the dialog box labeled as Grouping Variable, as seen in Figure 9.7.2. Next, you have to 
select Define Groups, which will tell SPSS what numbers you assigned to the two groups. 
Typically, this will be 1 and 2, but that is not always the case. If you wanted to use political 
ideology, for instance, you could compare the 1s and the 7s, those with the most strongly held 
positions as liberals and conservatives. In the example, we measured gender identity and had 
only two groups that emerged, male and female. Therefore, our grouping variable is “Gen- 
der” and we arbitrarily assigned males = 1 and females = 2 when we created the data file, 
so we enter 1 and 2 to define the groups (Figures 9.7.3 and 9.7.4). The language of “male” 
and “female” labels is more often associated with biological sex than with gender identity 
and therefore their use here is not ideal in terms of inclusive language. Next, we select the 
variable(s) that we would like to consider a dependent measure in this analysis from the entire 
list of variables. This is the variable(s) that we will explore to see whether males and females 
differed on that item/those items. It is possible to select more than one such Test Variable 
in this analysis, but in our example, we selected just one, an estimate of video game playing 
hours on a given day (Figures 9.7.5). Then you hit OK, and SPSS will run the test. In this 
example, we are testing the following hypothesis: 


H,: Those who identify as males will report more video game playing hours on a given day 
compared to those who identify as females. 


Figure 9.7.6 shows the SPSS output of that test, the independent t-test that measures 
whether the amount of video game use differs between those who identify as male and those 
who identify as female. The top part of the output, labeled Group Statistics, is the first place 
to look. It shows the mean number of hours of video game use among males and the mean 
number of hours of video game use among females, together with the corresponding standard 
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Figure 9.7.1 The Independent T Test command in SPSS 


Source: Stephen Warren 
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Figure 9.7.2 Select the grouping variable 


Source: Stephen Warren 
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Figure 9.7.3 Define the groups for the grouping variable 


Source: Stephen Warren 
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Figure 9.7.4 This tells SPSS that the first group was assigned a 1 in the data and the second group was assigned a 2 


Source: Stephen Warren 
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Figure 9.7.5 Select a dependent variable 


Source: Stephen Warren 


deviations. Just eyeballing those results, it sure looks like males outscore females on this esti- 
mate of video game use. Yet, to answer that question more persuasively, we look at the bot- 
tom part of the output. (Recall, as well, that N always means sample size. So, that part of the 
output tells us how many people in the sample identified as male and how many identified as 
female. You can also make note of the standard deviation numbers in the visual. You can see 
that there is less variability in the amount of time females reported spending on an average 
Friday compared to the amount of time males reported spending.) 

There is an intermediary step to take before we can conclude from the data that males 
report spending more time on an average Friday playing video games compared to females. 
The independent samples t test assumes that the variances of the two groups in question (in 
our case, males and females) are equal, but the test can correct for the possibility that they are 
not. The variance is similar to the standard deviation in that it has to do with how spread out 
values are in the data, but the difference between standard deviation and variance is that the 
variance is calculated from squaring the standard deviation. As you can see in Figure 9.7.7, 
SPSS will actually give you two different t results, two different degrees of freedom, and two 
different significance levels, one set corrected for unequal variances and one not. In order to 
determine which set of results to use—and this is the intermediary step we mentioned—you 
look at Levene’s Test for Equality of Variances. If the significance level of that test is p < .05, 
then you assume the variances in your two groups are unequal and you read and report the 
lower line for the t results. If the significance level for the Levene test is p > .05, you assume 
that the variances are equal and you read and report the upper line for the t results. 
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Figure 9.7.6 Examine mean scores for the two groups on the selected dependent measure 
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Figure 9.7.7 Output for the T Test 
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In the screenshot from SPSS for our example of an independent t-test measuring Friday 
video game use among males and females, we can see that the Equal Variances Assumed sec- 
tion of the output displays a statistically significant result, as we have indicated. Therefore, we 
must assume unequal variance in the two groups (males and females), which means that there 
is more or less dispersion/spread of values in one of those groups compared to the other (not 
surprising given the standard deviations we saw in the preceding result). Consequently, we 
read the t-test result from the bottom line of the output, as we have indicated. 

The most important parts of the output you obtain when SPSS runs a t-test are the f statis- 
tic, the degrees of freedom, and the p level. The degrees of freedom in a t-test are computed 
by taking the sample size and subtracting 1. If Levene’s test indicates unequal variances, how- 
ever, the degrees of freedom change from this standard. In any case, the convention when 
writing up the results from a t-test is to note the degrees of freedom in parenthesis after the 
t result. In our example, once again, the significance level for the Levene test (which SPSS 
displays as .000) is less than .05, and therefore, we report the lower line of results. Our result of 
t (310.59) = 9.18, p < .001, means that the likelihood of finding this difference in the amount 
of Friday video game playing between the males and females if it does not also occur in the 
population is less than 1 in 1,000. We would conclude, then, that the two groups are, indeed, 
different from one another on the variable in question. 

For both independent and dependent t-tests, it is difficult to interpret the meaning of the t 
result itself, except to say that generally, the bigger the t statistic, the more likely that you will 
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have a statistically significant result. This is logical because the larger the difference between 
the groups, the less likely you would observe a difference between groups in the sample that 
is not also present in the larger population. 

*IMPORTANT NOTE* You may have surmised that we are rounding to two decimal 
places when reporting our results in this chapter. That is, for the most part, the convention 
in the field. SPSS will often report that the significance level is .000. That does not actually 
mean, however, that the likelihood of observing the difference found in the sample when 
there is no such difference in the population is actually 0. It just means that the actual signifi- 
cance level is so small that it does not appear within the three decimal places that SPSS uses as 
the default. It is important to convert what appears in SPSS output as a p level of .000 as p < 
.001 (the smallest of the three main standards for statistical significance, as you will recall from 
Chapter 8) when this is the case. 


Tests of Differences: Analysis of Variance 


Some research designs require the researcher to look for differences on a variable of interest 
among three or more groups rather than two. For this type of statistical analysis, you need to 
run an analysis of variance test, abbreviated as ANOVA. There are many different types 
of ANOVA tests, but we will confine our discussion to what is called a one-way ANOVA, 
in which there is just one variable that organizes the sample into three or more groups for 
which differences on a variable of interest are tested. The ANOVA test produces an F statistic 
(this time signified by a capital F) and is calculated from the difference between the variability 
between groups and the variability within groups. 

Just like with t-tests, the grouping variable in ANOVA is typically a nominal-level variable. 
For example, you can test whether liberals, moderates, and conservatives differ from each 
other on a variable of interest. Political ideology, then, is the grouping variable in this example 
that divides the sample into three groups and is measured at the nominal level. The dependent 
variable, on the other hand, must be measured at the interval or ratio level, as is the case for 
t-tests, as well, so that the mean is meaningful. 

As you can see in Figure 9.8.1, this test can be found under the Analyze and then Com- 
pare Means tabs of SPSS. Similar to the t-test, you select a grouping variable to populate the 
bottom right of the dialog box, although this is called a Factor in this test. Also similar to 
the t-test you can select one or more variables to be the dependent measures to populate the 
upper right portion of the dialog box. You can see this in Figure 9.8.2, as well as see that 
you click OK once you have entered the variables. For our example, we have selected politi- 
cal ideology (with three groups: liberals, moderates, conservatives) as the factor and agree- 
ment with the statement “Boys should play with action figures, not dolls” (measured from 
1 = strongly disagree to 7 = strongly agree) as the dependent measure. The next step is to click 
on the Options box and then request descriptive statistics and hit continue, as is apparent in 
Figure 9.8.3. We take this step so that the output will include means and standard deviations 
for each group in addition to the statistics that show the likelihood of observing the differ- 
ences you see within those means if there were no such differences in the population. In this 
example, we are testing the following hypothesis: 


H,: Political conservatives are more likely than political liberals and moderates to believe 
that boys should play with action figures, not dolls. 


The final visual in this series, Figure 9.8.4, shows the SPSS output for this one-way ANOVA 
test. If we obtained a result like this, first, just like we did for the t-test, we would start by 
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Figure 9.8.1 The One-Way ANOVA command in SPSS 


Source: Stephen Warren 
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Figure 9.8.2 Select a grouping variable (factor) as well as a dependent measure 


Source: Stephen Warren 
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Figure 9.8.3 Select descriptive statistics to be included in the output via the Options button 


Source: Stephen Warren 


looking at the means. Again, agreement with “Boys should play with action figures, not 
dolls” was measured on a scale of 1 = strongly disagree to 7 = strongly agree, so higher means 
indicate more endorsement of this view. In these results, then, you can see that liberals were 
less likely to agree with this statement, followed by moderates, followed by conservatives, at 
least so it seems from the eyeball test. Once again, the bottom part of the output contains 
the significance level. Here p < .001, which indicates that there is a very low likelihood (1 in 
1,000) of observing the differences in means among people with the three different political 
ideologies that we see in the sample when there are no such differences in the population. 

Just like with the t statistic, it is difficult to interpret the meaning of the F result on its own, 
except to say that generally, the bigger it is, the more likely that you will have a statistically 
significant result. Once again, the logic is that the larger the difference observed in the sample 
among the groups, the more likely it is that those differences are also occurring in the popula- 
tion. There are two degrees of freedom reported in an ANOVA result, and they are reported 
in parenthesis after the F result, like this: F (2, 410) = 24.09, p < .001. The first degrees of 
freedom value is calculated from the number of groups in the analysis minus one. So, in this 
example, there were three groups in the analysis. The second degrees of freedom value is that 
first df number subtracted from the sample size. So, in this case, there was a sample size of 
n = 412. 

If we had a hypothesis that predicted that political conservatives would be more likely to 
think boys should play with action figures rather than dolls compared to political liberals, 
then, we could say that this hypothesis was, indeed, supported by the data. In the output, you 
can see that conservatives had a stronger endorsement of this view compared to liberals and 
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Descriptives 
Boys should play with action figures not dolls. 
95% Confidence Interval for 


Std. Mean 
Deviation Std. Error Lower Bound Upper Bound Minimum Maximum 
Liberal 1.863 -160 2.35 2.98 1 7 
Moderate 1.883 .136 3.57 4.10 1 7 
Conservative 1.806 -197 3.88 4.67 1 7 
Total 1.962 .097 3.35 3.73 1 7 
ANOVA 

Boys should play with action figures not dolls. 

Sum of 

Squares df Mean Square F Sig. 
Between Groups 166.830 2 83.415 24.087 .000 
Within Groups 1419.838 410 3.463 
Total 1586.668 412 


Figure 9.8.4 The means for the three groups and the result of the ANOVA test 


Source: Stephen Warren 


the chance of obtaining this result when there is no such difference in the population is less 
than 1 in 1,000. 


Tests of Relationships: Cross Tabulation with Chi Square 


A cross tabulation with chi square test is used to look for relationships between two (or 
more) nominal level variables. Rather than dealing with means and standard deviations, then, 
like t-tests or ANOVAs do, the chi square test looks at frequencies, comparing the distribu- 
tion that has been observed by the researcher in the data collected with distributions expected 
to occur by chance. Here is the notation for chi square: 7. 

In order to run a chi square test, the first step is typically to construct a cross tabulation 
table with cells in the table used to categorize the observed values (meaning, the values of 
the variables in question found in the data) of each of the two nominal level variables. As you 
can see in Figure 9.10.1, you will find this function in SPSS under Descriptive Statistics > 
Crosstabs. Doing so will produce the dialog box you see in Figure 9.10.2, in which you select 
the variables you would like to use for the test. If there is a variable that you would consider 
to be the independent or predictor variable, the convention is to place that variable in the 
column position in the bottom box of the upper right portion. You place the variable that you 
would consider to be the dependent variable in the row position in the upper box. Next, you 
click on Statistics and then OK (again, as seen in Figure 9.10.2). This will produce a dialog 
box with many options. For our purposes for this example, we would select Chi-square, as 
indicated in Figure 9.10.3 and then hit Continue. Before we ask SPSS to actually run the test, 
there is one more intermediary step that we advise you take. You should select the Cells tab 
(Figure 9.10.4) and check off Column under Percentages (Figure 9.10.5), which will produce 
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not just counts/frequencies in each cell of the cross tabulation table but the percentage of 
respondents who chose each option on the variables in question. Then hit Continue, which 
will actually perform the test and produce the output. 

Figures 9.10.6 and 9.10.7 show SPSS output associated with a cross tabulation with chi 
square test of whether respondents’ sexual orientation was associated with their political ideol- 
ogy (again, two nominal level variables). In the cross-tabulation table (Figure 9.10.6), the top 
left cell represents the number of people (and percent of the total sample size) that identified 
as straight/heterosexual and as liberal. That is what a cross tabulation does: It shows the over- 
lap between two variables. In fact, the entire top row shows numbers of people (and, again, 
percent of sample size) that identified as liberal across all of the sexual orientation possibilities 
in the sample. The first column (also circled) shows the distribution of those who identified 
as straight/heterosexual across all of the political ideology categories. 

In order to determine whether the two variables are related to each other, you begin with 
an eyeball test of the distribution of the values, and then you look to the second part of the 
output. The first stage in conducting a test like this is to construct a cross-tabulation table and 
look for an imbalance among the cells. If the two variables were not related, there would be a 
relatively equal distribution across each of the cells of the table rather than the more lopsided 
numbers. Just from the eyeball test, then, it does seem that sexual identity and political ideol- 
ogy are related. 
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Figure 9.10.1 The Crosstabs command in SPSS 
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Figure 9.10.2 Enter variables for the test and select Statistics 
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Figure 9.10.4 Select the Cells button 
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Figure 9.10.5 Select Column percentages 


Source: Stephen Warren 
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Chi-Square Tests 


Asymptotic 
Significance 
Value df (2-sided) 
Pearson Chi-Square 16.499° 6 .011 
Likelihood Ratio 17.741 6 .007 
Linear-by-Linear 7.446 1 .006 
Association 
N of Valid Cases 413 


a. Five cells (41.7%) have expected count less than 5. The 
minimum expected count is .81. 


Figure 9.10.7 Output for the Chi-square test 


Source: Stephen Warren 
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To find out whether the relationship between variables that you have speculated about 
by looking at the distribution in the table is statistically significant, you report from the very 
top line of the portion of the output labeled Chi-Square Tests (Figure 9.10.7). The results 
show ¥? = 16.50, df= 6, p = .01, and you would report them just like that. Just like t and F 
statistics, y? is difficult to interpret on its own, although larger values are usually associated 
with a statistically significant p level. The dffor chi square is calculated by the number of cells 
minus 1 multiplied by the number of rows minus 1. The p value in our output is the most 
important component, then. It says that the odds of observing the association between sexual 
orientation and political ideology present in our results when there is no such association in 
the population (the chance that there is actually equal distribution across cells rather than the 
imbalance across cells that we found) is equal to 1 in 100. Thus, we can safely conclude that 
there is an association between these two variables, and the association has a particular direc- 
tion: LGBTQ+ people are more likely to identify as liberal compared to as conservative. We 
made that observation by looking at the cells for all those who did not identify as straight and 
seeing whether they clustered more closely around liberal ideology rather than being more 
spread out across differing political ideologies. 

*IMPORTANT NOTE* The math behind chi square tests is thrown off if you have too 
many cells with not very many counts in them (too many underpopulated cells). That is why 
there is a footnote associated with the chi square result in our output. The rule is that no more 
than 20% of the cells should have an expected frequency of five, whereas we have 42% in our 
example. Before we place too much value on this result, then, we would need to correct this 
problem. 

There are two ways of doing so. First, you could recode variables (using the process we have 
explained in this chapter) to collapse categories so that they now have more counts in them. 
This could certainly be done for the sexual orientation categories since some have so few. Or 
you could run a different test that is not sensitive to this issue, such as Fischer’s exact test of 
independence. It would be good for you to practice SPSS use now with the sample data set 
by (1) recoding the sexual orientation variable into a new variable with fewer categories and 
(2) running a new cross tabulation with chi square test to see if the 20% or fewer cells with a 
minimum expected frequency of five issue has been resolved when using the newly recoded 
sexual orientation variable in the test for its association with political ideology. 


Tests of Relationships: Pearson’s Correlation 


The next major type of bivariate and multivariate statistical tests is used to see whether two 
or more variables relate to one another. If variables are found to relate—with “relate” defined 
as whether knowing the values of one variable helps predict the values on the other—we say 
that they are correlated or associated with each other. If they are found not to relate, we say 
that they are independent of each other. 

If you would like to see whether two variables are related to one another and both of those 
variables are measured at the interval or ratio level, you run a Pearson’s correlation test. 
The Pearson’s correlation test determines whether changes in the values of one variable are 
linked to changes in the values of another variable. It does so by calculating sums of values 
on each of the two variables in question, squaring those sums, and calculating the product 
of those sums. But again, knowing this equation is sort of like knowing how factories make 
sausage. You do not really have to know how to calculate correlation by hand, and perhaps, at 
this point in learning quantitative methods, you do not want to know. It is more important in 
an introductory quantitative methods context to simply know how to run this test using SPSS 
and to know what the output of the test means. 
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A Pearson’s correlation test calculates a correlation coefficient, abbreviated with a low- 
ercase r. That coefficient ranges from -1 to +1. The sign in front of the coefficient signifies 
whether the relationship between the two variables is negative or positive. A negative cor- 
relation means that as the values of one variable increase, the values of the other decrease (we 
can also say that these two variables are inversely related). A positive correlation means that 
as the values of one variable increase, the values of the other variable increase, as well. 

The value of the coefficient represents the strength of the relationship. The closer the coef- 
ficient is to 0, the weaker the relationship between two variables. The closer the coefficient 
is to either -1 or +1, the stronger the relationship. Indeed, if you were to find two variables 
with a correlation coefficient of 1.0, that would mean that the two are perfectly correlated. 
Every time a value on one variable increases, a value on the other increases, as well. (The 
same applies to a correlation of -1.0, except in this case the two would be perfectly negatively 
correlated.) We should mention, however, that this result is much more likely to be a feature 
of the natural world than the social world. The social world is just too diverse and complex 
to have two variables be perfectly correlated. 

To run a Pearson’s correlation in SPSS, you go to Analyze Correlate Bivariate, as you 
can see in Figure 9.11.1.1. You select any two or more interval or ratio level variables that you 
would like to test to see whether they are correlated from the list of variables in the data set, as 
you can see in Figure 9.11.1.2. In our case, we have selected Average hours playing video games 
per day and, once again, agreement with the statement, “Boys should play with action figures, 
not dolls’? You will also select Pearson under Correlation Coefficients, as we have indicated 
with an arrow in Figure 9.11.1.2. Then you hit OK and SPSS will run the test. 

Figure 9.11.1.3 shows the SPSS output associated with this test. SPSS produces what is 
called a correlation matrix, and the diagonal of that matrix always shows each variable 
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Figure 9.11.1.1 The Bivariate Correlation command in SPSS 


Source: Stephen Warren 
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Figure 9.11.1.2 Select the variables of interest and the Pearson's test 


Source: Stephen Warren 
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Figure 9.11.1.3 Output for the Pearson's correlation test 


Source: Stephen Warren 
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correlated with itself. That is why you see the 1s on the diagonal. Of course, the video game 
use variable correlates perfectly with itself, as does the “boys should play with action figures” 
variable. But the answer to the question of whether the two correlate with each other is 
located in the other cells of the matrix and is circled in Figure 9.11.1.3. In this case, we would 
report r = .04, ns. The correlation between the amount of video game use and agreement 
with this statement is quite close to 0. The significance level is way over .05, which means that 
there is no significant relationship between those variables. If we had a hypothesis that pre- 
dicted that amount of video gaming would be associated with this view of masculine gender 
roles, then, we would have to reject that hypothesis. 

Unlike the t or the F statistic, the r coefficient is quite meaningful on its own since its size 
(or distance between 0 and either -1 or +1) determines how weakly or strongly the two vari- 
ables are related. This chart indicates how researchers typically discuss the size/strength of a 
correlation coefficient. 


Coefficient Interpretation 

0 no relationship 

.1 to .3 weak relationship 

4 to .6 moderate relationship 
.7 to .9 strong relationship 
1.0 perfect relationship 


There is another helpful property of a correlation coefficient, as well. If you square the value 
of the coefficient, you can calculate what is called the coefficient of determination, which 
tells you how much variation in one variable is explained by the variation in the other vari- 
able. The correlation coefficient, r, becomes 7’. And r tells us how much of Variable A is 
explained by Variable B. So if age and earnings, for example, have a correlation of r = .60, 
then r = .36, which means that we can say that 36% of the variation in the earnings variable 
is explained by variation in the variable age. Logically, that means that 64% of the variation in 
earnings is not explained by age. Clearly, some other variables also contribute importantly to 
explaining earnings. This is an important cautionary remark about the fact that there is always 
more to the story than what any single statistical analysis can tell you about a phenomenon 
of interest. 

Pearson’s correlation analyses can also be run with what we call a control variable, deter- 
mining whether any relationship first found between two variables will hold up when intro- 
ducing a third. Recall from the chapter on experiments that a control variable is defined as 
any variable held constant in an analysis so that a relationship between two other variables can 
be clarified or confirmed. Imagine we find a correlation between the amount of ice cream 
consumed and the number of burglaries or other crimes committed in a study, for instance. If 
we enter a third variable, outside temperature, into the equation, we are likely to see any ini- 
tial correlation between ice cream consumption and crime disappear. That initial relationship 
between eating ice cream and crime was spurious. It was explained away by the introduction 
of a third, control variable: hot weather. Hot weather is a cause of ice cream consumption as 
well as an underlying cause of crime. 

The next set of SPSS output visuals shows how to run Pearson’s correlation with a control 
variable. To do so, you select Analyze— Correlate Partial from the pull-down menu, as 
seen in Figure 9.11.3a. Then you populate both parts of the dialog box, entering the two 
variables that will be tested for correlation in the top and the control variable in the bottom. 
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Figure 9.11.2 A reminder that correlation is not the same as causation, partly because a third variable may 
be at play 


CAUSATION 


Source: Rcragun, https://commons.wikimedia.org/wiki/File:Correlation_vs_causation.png 


As you can see in Figure 9.11.3b, we have elected to determine whether the amount of time 
spent playing video games correlates with the amount of time spent watching TV while 
controlling for age, using these variables from the sample data set. Figure 9.11.3c shows the 
SPSS output for this test, a correlation matrix labeled to indicate that it reflects the correla- 
tion between the two variables in question while controlling for the third. How would you 
interpret the result, using the components of the output we have circled? Are these two forms 
of media use correlated with each other, even when accounting for age of the respondent? 


Tests of Relationships: Multiple Regression 


Regression tests are also quite flexible and potentially powerful statistical tests. In a regres- 
sion test, one or more variables are entered into a statistical model as predictors of a single 
dependent variable. In other words, regression tests are used to explain how much of the 
variation in a dependent variable is explained as a function of the variation of one or more 
independent/predictor variables. When there is more than one predictor/independent vari- 
able, we call these multiple regression tests. Multiple regression is the test to use if you would 
like to explore whether and how much multiple variables predict scores on a single dependent 
measure. 

In multiple regression, the dependent variable must be measured at the interval or ratio 
level and the predictors are typically measured at those same levels, as well. If the predictor 
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Figure 9.11.3a The Partial Correlation command in SPSS 


Source: Stephen Warren 
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Figure 9.11.3b Select variables of interest, including a control variable 


Source: Stephen Warren 
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Figure 9.11.3c Output for Partial Correlation test 


Source: Stephen Warren 


variables are not interval or ratio, they must be transformed before they can be entered into 
a regression model. To transform a nominal level variable to use as a predictor in multiple 
regression, for instance, you must create a dummy variable, as we explained early on in this 
chapter. In a dummy variable, one of the values of the variables is used as a reference group 
and is given a score of 1, and the other values are all given a score of 0. Multiple regression 
tests tell you how much each predictor variable entered into the model contributes to the 
dependent measure, one by one, as well as how much they contribute together. 

Stepwise regression is a type of multiple regression in which the researcher determines 
the order in which to enter predictor variables, with each set of predictor variables entered 
in steps. When the first block of predictor variable(s) is entered, the test will give you a 
R? value, which is measured from .0 to 1.0 and tells you how much of the variance in the 
dependent measure is explained by the first block of predictors. When subsequent blocks are 
entered, the test tells you how much additional variance in the dependent variable—above and 
beyond that explained by the first block—each subsequent block explains. The first block of 
variables often consists of demographic variables—like gender, age, race, ethnicity, etc. The 
next block(s) can be any other variable or variables of interest that are thought to predict the 
dependent measure. The final block is usually reserved for the hypothesized predictor, because 
when this block is entered into the regression equation, the equation is still accounting for all 
other blocks that came before it, so the results will tell you once again how much additional 
variance in the dependent measure is explained by that final predictor even after accounting for 
all the variables already entered. It is a rigorous test of a hypothesis given that the last predictor 
entered has to still explain some of the variance in the dependent measure even after account- 
ing for all of the variables that have been entered into the regression equation before it. 

Figures 9.12.a through 9.12g show how to run a stepwise regression test in SPSS, as well as 
what components of the resulting output you report. As you can see in the first visual, you can 
find the test under Analyze Regression—Linear. The next step is to plug in the dependent 
measure in the top portion of the resulting dialog box, and then to enter the first variable 
or variables for the first block, followed by clicking on Next, as you can see in the second 
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visual. In our example from the sample data set, we are entering predictors to see how well 
they help explain the dependent variable of level of agreement with the notion that dominant 
masculine roles and norms emphasize Restrictive Emotionality. For Block 1, we have chosen 
to enter two demographic variables, age (a ratio level measure) and political ideology (meas- 
ured on a seven-point scale, and therefore considered interval). In Block 2, we have entered 
our hypothesized predictor, amount of time spent playing video games per day (please see the 
third visual in this series). Since we are only using two blocks of predictor variables in this 
example, after entering that one, we can click OK at the bottom right rather than Next. If 
you were to click Next, you could enter a third block of predictors. The next step is to select 
the Statistics tab and check off R? change, as we have indicated with the arrow in the fourth 
visual (Figure 9.12d). Then, you click Continue and SPSS runs the test. 

The output is shown in the last visuals in this series. The Variables Entered/Removed por- 
tion of the output on the top left simply summarizes the stepwise structure used to run the 
test (Figure 9.12e). The most important parts of the output appear in the Model Summary 
in the fifth visual (also Figure 9.12e). The row that says Model 1 shows how much of the 
dependent variable was explained by the first block of predictors, in our case age and political 
ideology. First, you look at the R? portion of that table, as we have circled, in addition to the 
Sig. F change value associated with that first R? result. Here we see that adding age and politi- 
cal ideology to the regression equation resulted in an R? of .077, which was significant at the 
p < .001 level. Age and political ideology, then, are, indeed, significant predictors together 
in the same block of views of Restrictive Emotionality in masculine gender roles and norms. 
The next row in the Model Summary, Model 2, shows what happened when the amount of 
time spent playing video games was entered into the regression equation. The R? associated 
with that variable was .099, and the significance level associated with that result was .002, as 
we have circled. Since .002 is a statistically significant finding, we would say, then, that video 
game playing was also a significant predictor of views of Restrictive Masculinity even after 
taking age and political ideology into account. The last important part of the Model Sum- 
mary table in the third visual is R? change. This shows that adding the second block—which, 
once again, was the video game playing variable—increased the amount of variance in views 
of Restrictive Masculinity by .022. This makes sense since the first block (age and political 
ideology) had an R? of .077, and the second block (video game playing) had an R? of .099. 
The difference between those two values is the R? change of .022 (.099 minus .077). 

Regression tests are based in part on analysis of variance, and the second visual from the 
output (the sixth one in this series, Figure 9.12f) shows the ANOVA calculations for each of 
the two models. You can see that these same F results are apparent in the Model Summary 
from the first output visual we have already explained. The final important aspect of the find- 
ings stems from the final visual in this series, labeled Coefficients, Figure 9.12g. This part of 
the SPSS output shows not just what the entire block contributed to the dependent variable 
but what each variable within each block contributed. It does so by providing Beta coef- 
ficients (abbreviated B) for each variable, which are measures of the strength of the effect of 
entering each variable into the equation, how strongly the variable helps explain the depend- 
ent measure. The higher the Beta coefficient, the stronger the predictor, and each Beta coef- 
ficient also has its own associated significance level. 

Some people report standardized Betas in their results, others report unstandardized Betas, 
and still others report both. Unstandardized Betas are those produced by the raw data. There- 
fore, they are measured in the same units as the original measures. Standardized Betas, on the 
other hand, are based on z scores, which are transformations of raw data in which variables 
are changed to have a mean of 0 and a standard deviation of 1, no matter what the units 
among those variables were. So, which Betas should you use in your research? The advantage 
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Figure 9.12a The Linear Regression command in SPSS 


Source: Stephen Warren 
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Figure 9.12b Enter the first block of variables 


Source: Stephen Warren 
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Figure 9.12c Enter a subsequent block of variables 


Source: Stephen Warren 
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Figure 9.12d Select R squared change so that it is included in the output 
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+ Regression 


Variables Entered /Removed* 


Variables Variables 
Model Entered Removed Method 


1 War is your . Enter 


2 Average . Enter 


ng 
Video Games 
per day 


a. Dependent Variable: MRNI Restrictive 
Emotionality Subscale 


P aay cor. | 


Model Summary 


Change Statistics 


Adjusted R Std. Error of R Square 
Square the Estimate Change F Change dfl df2 Sig. F Change 


1.24136 
1.22790 


a. Predictors: (Constant), 


b. Predictors: (Constant), What is your political orientation?, Before we begin, please give your age in years, Average hours 
playing video games per day 


Figure 9.12e Output for the Linear Regression test, model summary 


Source: Stephen Warren 


ANOVA? 


Sum of 
Model Squares df Mean Square F Sig. 


1 Regression 
Residual 631.803 410 1.541 
Total 684.654 412 

2 Regression 67.984 3 22.661 
Residual 616.669 409 1.508 
Total 684.654 412 


a. Dependent Variable: MRNI Restrictive Emotionality Subscale 


b. Predictors: (Constant), What is your political orientation?, Before we begin, 
please give your age in years 


c. Predictors: (Constant), What is your political orientation?, Before we begin, 
please give your age in years, Average hours playing video games per day 


Figure 9.12f Output for the Linear Regression test, F statistic and other ANOVA results 


Source: Stephen Warren 
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Coefficients* 


Standardized 
Unstandardized Coefficients Coefficients 


Model B Std. Error Beta t 
1 (Constant) 

Before we begin, please 

give your age in years 

What is your political 

orientation? 
2 (Constant) 


Before we begin, please 
give your age in years 


What is your political 
orientation? 


Average hours playing 
video games per day 


a. Dependent Variable: MRNI Restrictive Emotionality Subscale 


Excluded Variables? 


Collinearity 

Partial Statistics 

Model Beta In t Sig. Correlation Tolerance 
1 Average hours playing 1495 3.168 .002 .155 .997 


video games per day 
a. Dependent Variable: MRNI Restrictive Emotionality Subscale 


b. Predictors in the Model: (Constant), What is your political orientation?, Before we begin, 
please give your age in years 


Figure 9.12g Output for the Linear Regression test, Beta coefficients 


Source: Stephen Warren 


of using unstandardized Betas is that they are easier to interpret since they mirror the way 
that the variable(s) was measured. Yet, it is considered best practice to report standardized 
betas when you are comparing predictors that were measured using different units, since 
the z scores give them a common metric and therefore facilitate a comparison across them. 
Many researchers choose to report both. We have emphasized the unstandardized Betas in 
Figure 9.12.g, as you can see by the circle. Reading that output, then, we would report that 
age (B = .08, p < .01) is a significant predictor in the final model (Model 2), as is political 
ideology (B = .19, p < .001). Importantly for the hypothesis, amount of video game playing 
is also a significant predictor of views of Restrictive Emotionality in the final model (B = .07, 
p < .01). When running regression tests for your own research, you would report the Beta 
coefficients, the R’, and the change in R? that happens from block to block when each block 
is entered in the equation. 


Summary 


We have taken you on a whirlwind tour of some of the most commonly used statistical tests 
in Communication and other social sciences. It is a lot to take in, we know, and we hope we 
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have set out the steps in a logical and easy-to-follow manner. Even though we have covered 
a great deal for an introductory book, we must confess that what we have explained in this 
chapter is only the tip of the iceberg. As you saw from the non-highlighted components of 
the SPSS commands we have illustrated, there are loads of other options of tests to run and 
ways to analyze your data using SPSS (or any other statistical package). We hope that this will 
be just the beginning of your own exploration of the statistical tools and approaches available 
to make sense of quantitative data. 

In this chapter, we have introduced measures of central tendency and measures of variability 
and shown you how to run those tests on your own variables in SPSS. We presented a handful 
of the most frequently used ways to transform variables, including computing new variables 
from those that already exist, recoding variables, selecting cases, and creating an index (includ- 
ing first determining through a reliability analysis whether an index is warranted). Next, of 
course, we got into the details of tests of differences between and among groups and tests of 
associations between and among variables. When first learning statistics, it can be quite daunt- 
ing to have to determine what test(s) you should run to explore your research questions and 
test your hypotheses. In this chapter, we have shown that selecting a test depends a great deal 
on what the data themselves look like, including what level of measurement each variable uses 
and, in some cases, how many groups are created by variables in the data. Although there may 
seem to be many ways to analyze data, then, the decision about how to begin is often quite 
straightforward, since the properties of the variables dictate the statistical approach. 

If statistics really are the grammar of science, as statistician Karl Pearson said, then the best 
way to learn them is to use them, just like any other new language that you might learn. 
It certainly takes time and practice to internalize why and how you approach data analysis 
in particular ways, and we encourage you to keep up with that practice as your interests in 
quantitative research grow. We would also remind you of the link between Chapter 8 and 
this chapter, and say once again that it is just as important to know why you are running a 
particular statistical test as it is how to run it. 

And it is certainly equally as important to know what to make of the results that you find. 
Interpretation is key. It is the way in which we communicate what we found, what it means, 
and why it is important. It is important not to get lost in the detail or in the mechanics of 
statistical analysis but rather to remember that the numbers you are crunching represent real 
people and/or real topics of real importance to the world. 

In order for quantitative research data to be of use in social justice efforts in the wider 
world, it is vitally important that results of studies are clear, accurate, and precise. Given that 
not all readers of your research will be trained in statistics, the onus is on you to communicate 
your results in a manner that is both statistically defensible and readily understood. In the last 
chapter of this book, we discuss how to write up quantitative research for presentation, pub- 
lication, for use in policy formation, and for the wider public. 


Discussion Questions and Activities 


1. Imagine you are an interpersonal communication scholar with an interest in family com- 
munication. You have a hypothesis that predicts that the more one’s family typically 
engages in argumentation at the dinner table (measured by the approximate number of 
times per month), the more likely one is to report intention to vote in the next election 
(measured by the degree of likelihood on a scale of 1 to 7). What statistical test would run 
to test this prediction? Why did you choose that test? 

2. Imagine that you are a health communication scholar running an experiment to test par- 
ticipants’ responses to racial diversity among the characters appearing in a public service 
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announcement (PSA) reminding people to get a flu vaccine every year. You have three 
groups in your research design: those who see a PSA featuring an African American fam- 
ily, those who see a PSA featuring a White family, and those who see no PSA at all. You 
want to know whether those different types of exposure influence intention to get a flu 
shot this year among your research participants, which you measured as a Yes or No item. 
What statistical test would you run to find out the answer to your research question? Why 
did you choose that test? 

3. Now imagine a slight change in the previous scenario. Rather than testing whether par- 
ticipants intend to get a flu shot, you are now interested in how important they think 
getting a flu shot is, on a scale of 1 to 5. What statistical test would you run to find out 
this answer? Why did you choose that test? 

4. From your own research interests, come up with a hypothesis that you would like to test, 
a way to operationalize the variables you would need to test that hypothesis, and then a 
data analysis plan for the statistical tests you would use to find out whether your hypoth- 
esis was supported or refuted. 
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10 Mixed Methods 
Triangulating with Qualitative Research 


As we have seen in Chapter 1 of this book, social justice researchers often take a problem-cen- 
tered approach to research with the goal of achieving social transformation. Given this goal, 
sometimes quantitative research methods alone are not sufficient to fully understand, explain, 
predict, challenge, and change social inequalities. Within social justice research, it is quite 
common to combine quantitative methods with qualitative methods to approach the research 
problem from multiple perspectives. Combining multiple research methods, also known as 
a “mixed methods approach,” is especially useful when addressing complex communication- 
related social issues. 

Although we have examined quantitative methods such as experiments, surveys, and con- 
tent analysis in separate chapters in this book thus far, these methods can be used in con- 
junction with one another within a larger research project. Using different types of methods 
within the same research tradition is called a multi-methods approach. Thus, multi-meth- 
ods approaches use multiple quantitative methods (or multiple qualitative methods) within a 
research project. For example, let’s say that a quantitative researcher is interested in under- 
standing anti-Chinese racism in the U.S. during the COVID-19 pandemic. They could 
potentially conduct a quantitative big data analysis of a hashtag such as #lamNotaVirus fol- 
lowed by a national survey of Asian Americans about their social media use to discuss the rac- 
ist experiences they have encountered. Similarly, researchers Riva Tukachinsky, Dana Mastro, 
and Moran Yarchi (2017) used a multimethod quantitative approach to examine the effects 
of television representations of Black and Latinx characters among U.S. viewers. They first 
conducted a content analysis of primetime shows to note the patterns in representations of 
these groups. These findings were then used as predictors in the second stage of the project. In 
the second stage of the project, the researchers analyzed cross-sectional national survey data to 
examine whether exposure to these TV shows was correlated with viewers’ attitudes toward 
Latinx and Black individuals during the same time period. 

Whereas multi-method approaches use multiple quantitative or multiple qualitative meth- 
ods within the same project, mixed methods approaches are slightly different. Mixed meth- 
ods is a term specifically used for research that involves both qualitative and quantitative 
research within the same project. Mixed methods research is an approach where “a researcher 
or team of researchers combines elements of qualitative and quantitative research approaches 
(e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference tech- 
niques) for the broad purposes of breadth and depth of understanding and corroboration” 
(Johnson, Onwuegbuzie, & Turner, 2007, p. 123). As we have seen in Chapter 1, qualitative 
and quantitative research methods have slightly different assumptions, values, and orientations. 
By complementing one another, both approaches provide different but helpful perspectives 
on social issues. In general, quantitative research focuses on measuring variables, understand- 
ing statistical relationships among them, testing hypotheses, explaining relationships, and 
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predicting outcomes. In contrast, qualitative research emphasizes in-depth analysis, subjective 
evaluations, and meaning-making. Depending on the goals of a research project and available 
resources, social justice researchers can use a single method, multimethod, or mixed methods 
approach to examine social issues. 

The rest of this chapter provides a framework for social justice researchers interested in 
using mixed methods approaches that combine quantitative with qualitative methods. We 
will first discuss the concept of triangulation, which helps to understand the significance of 
approaching research issues from multiple viewpoints. Then, we will introduce you to some 
common qualitative methods used in Communication research, especially for social justice. 
Toward the end of the chapter, we explain how, why, and when to use mixed methods 
research for addressing social justice issues within Communication. 


Triangulation: Approaching Research from Multiple Perspectives 


Triangulation is a term used to describe the combination of multiple approaches and per- 
spectives within the same study to understand the same phenomenon. It is a commonly 
used strategy that allows for describing, analyzing, and decoding phenomena from multiple 
perspectives using multiple methods. It allows researchers to access multiple meanings, reali- 
ties, and viewpoints in addressing a social issue. Triangulation is particularly helpful in social 
justice-related social problems that involve multiple stakeholders and long histories of injus- 
tices. Bringing in multiple research perspectives could help to address these social issues in 
effective and innovative ways. 

There are many ways in which triangulation can be achieved, by combining data, researcher, 
theoretical, and methodological approaches (Denzin & Lincoln, 2005). Data triangulation 
can be achieved by gathering data about the issue or topic across time, space, and populations. 
Collecting data using longitudinal designs, for instance, helps researchers to examine the 
issue across time. Similarly, data collected across various subgroups and cultures also provides 
multiple perspectives. Researcher triangulation involves the practice of having multiple 
investigators and researchers examining the same social issue. For example, in Chapter 7 on 
content analysis, we have seen that intercoder reliability is assessed by having multiple cod- 
ers examine the same texts to understand the extent of similarity in their interpretations. 
Theoretical triangulation is when a researcher examines the issue of interest using multiple 
theoretical perspectives to gain new insights. For instance, a researcher studying everyday sex- 
ism might examine it using two different theoretical approaches. Finally, methodological 
triangulation is when we use multiple methods within the same project to study a topic 
or an issue. By using multiple methods, slightly different types of questions can be addressed 
within the larger research purpose of the project. Perhaps the social justice research includes 
surveys and experiments. Or the researcher includes experiments and content analysis within 
their project. A key goal of methodological triangulation is to increase the validity of the phe- 
nomenon being studied by measuring and observing it from multiple methods, standpoints, 
and perspectives. 

For the purposes of this chapter, we will focus on one type of methodological triangula- 
tion—the use of mixed methods within the same project. Mixed methods research combines 
quantitative and qualitative methods (Figure 10.1). Mixed methods projects could involve any 
quantitative method (such as content analysis, surveys, and experiments) along with any of the 
diverse range of qualitative methods (such as case studies, ethnography, interviewing, focus 
group discussions, textual analysis, and arts-based research, to name a few). Mixed method 
research helps researchers validate their findings, strengthen their conclusions, address con- 
tradictions, and include participants’ voices in the research process. This approach facilitates 
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Figure 10.1 Mixed methods: triangulation by combining quantitative and qualitative methods 


Source: Emily Riewestahl 


greater collaboration and cooperation among researchers. However, some of the drawbacks 
of mixed methods research are that they could involve complex designs, may require large 
samples, be time-consuming, need additional funding, be labor-intensive, and involve large 
multidisciplinary research teams. 


Mixed Methods Approaches in Communication 


Mixed methods continue to be quite rare within published studies in Communication, 
where the norm is to use only one of the two sets of methods. Communication is often 
seen as interdisciplinary in nature, and there is a rich history of diverse methodologi- 
cal perspectives used across the qualitative and quantitative continuum. However, these 
strands of research have typically grown separately alongside each other, rather than being 
used together in the same project. While purists among both quantitative and qualitative 
researchers might say that quantitative and qualitative methods are like oil and water that 
never mix, combining these methods could be a good way to balance the strengths and 
weaknesses of both approaches. As we have seen in Chapter 1, quantitative and qualitative 
methods are both valid and helpful in addressing communication problems. It is important 
to understand their respective strengths and weaknesses so that they are employed appro- 
priately at different stages of the research design process to address issues of concern to the 
researcher. 
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A key moment within Communication was the “paradigm wars” of the 1970s and 1980s 
between quantitative social scientists who valued positivism and scholars from critical cul- 
tural studies who used interpretive and social constructivist approaches to Communication 
research. (You may want to revisit Chapter 1 to recall the distinguishing features of these dif- 
ferent paradigms.) In describing the relationship between quantitative and qualitative methods 
within Communication, DeCoster and Lichtenstein (2007) explain that “The quantitative- 
qualitative divide is sometimes stark, angry, or dismissive” (p. 228). While the so-called quan- 
titative-qualitative divide is no longer quite as fraught and stark, there continue to be tensions 
and misperceptions from both sides. However, we believe that qualitative and quantitative 
research can complement each other and work synergistically and interactively in the study 
of Communication, especially as it relates to social justice issues. Approaching qualitative and 
quantitative research in inclusive and respectful ways can help bring about meaningful social 
transformation (Mertens, 2007, 2010). 

While it is still not very common to combine qualitative and quantitative methods within 
the same research project in Communication, there is a growing realization that doing so 
could be beneficial to gaining deeper and broader insights, especially to address complex 
social issues (sometimes also known as “wicked” problems). For instance, within health com- 
munication, scholars such as Mohan Dutta (Dutta, 2008, 2018) have combined quantitative 
methods such as field experiments and surveys with many of the qualitative approaches that 
will be defined and discussed in this chapter—such as focus group discussions, participant 
observation, and ethnography—in community-driven research projects. Within political 
communication contexts, scholars such as Genevieve Johnson and colleagues (2019) have 
combined surveys and participant observation to examine emotions expressed by partici- 
pants in citizens’ groups focused on political engagement. Some media scholars, too, have 
used mixed methods approaches. For example, Anita Atwelle Seate and colleagues (2016) 
have used rhetorical analysis and experimental research to examine nationalistic themes in 
the promotional ads for the 2014 World Cup soccer match between the United States and 
Ghana. 


An Introduction to Qualitative Methods in Social Justice Research 


Qualitative methods have been central to social justice projects, providing unique insights 
and perspectives, especially in community-based projects. As we have seen in Chapter 1, 
qualitative methods are the dominant modes of inquiry for studying and building on critical 
cultural communication, feminist theory, queer theory, postcolonial theory, and critical race 
theory. Critical cultural approaches within Communication have relied heavily on qualitative 
methods to examine research questions. There is a rich and long history of the use of diverse 
qualitative methods in approaching social, political, and cultural issues, especially relating to 
race, gender, sexuality, nationality, and other social justice topics. 

Recall from Chapter 1 that there are two main paradigms within qualitative research: the 
interpretive or constructivist paradigm and the critical paradigm. The interpretive or con- 
structivist paradigm examines the processes involved in sense-making and (re) constructing 
meanings through everyday practices, events, situations, texts, and experiences in communi- 
cative interactions. Drawing from theories such as symbolic interactionism, phenomenology, 
and dramaturgy (Leavy, 2017), this paradigm emphasizes that meanings are not static and that 
shared meanings are constantly evolving through interactions. The second main paradigm that 
informs qualitative research is the critical paradigm. As critical social researchers interested 
in social justice issues, this paradigm is particularly relevant for this book. As we have seen in 
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earlier chapters, the emphasis of the critical paradigm is in understanding power differences, 
social inequalities, and structural hierarchies. It examines how power is maintained, negoti- 
ated, and resisted in various contexts, especially through justifying dominant ideologies as 
commonsense and as a given mode of doing things. As critical Communication researchers, 
qualitative research is helpful to understand how power operates within the symbolic contexts 
in which people are immersed. 

Whereas the purpose of quantitative methods is to seek generalizable explanations for social 
behaviors and phenomena, qualitative research often provides rich description of complex 
phenomena that is narrower in scope, which cannot be conveyed quantitatively through num- 
bers. Given the emphasis on aggregated data, quantitative research methods cannot be tai- 
lored to individual needs and contexts, thus making it hard to capture nuances and intricate 
details of experiences, which would be classified as outliers and irrelevant to the study from 
a quantitative perspective. Qualitative analyses help researchers understand the variations and 
complexities of everyday phenomena in their natural environments. 


Figure 10.2 Fieldwork in a naturalistic setting 
Source: Photo by Hello Pm Nik on Unsplash, https://unsplash.com/photos/YiR QlglwYig 
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The qualitative researcher uses emergent design principles, where the study can and 
almost always does change as the research process unfolds (Creswell, 2014). Rather than have 
fixed hypotheses and research procedures predetermined, there is flexibility in the research 
design. The researcher can also feel free to introduce new factors or procedures into the 
design as they deem appropriate. It tends to be more inductive in its approach such that the 
research design and the interpretation of the observations can evolve and develop throughout 
the research project. Hypotheses are not predetermined. Rather, observations about what 
is observed evolve through the process of the research study. Using an inductive approach 
means that theory-building is based on deep listening to the data that has been collected, 
including more subtle meanings that may not be readily apparent. For instance, Bates and 
colleagues (2019) describe how they engaged in a health communication project over a three- 
year period in Ecuador using focus groups, mapping of community resources, and in-depth 
interviews with three communities that involved dialogue, discussion, and drawing exercises. 
Many aspects of this research design were not predetermined but emerged as the study pro- 
gressed. Even in social justice research that is not community-based, emergent design allows 
for greater flexibility in evolving research procedures and interpretations of the meanings as 
new data are collected and analyzed. 

Qualitative methods focus on naturalistic inquiry, with an emphasis in capturing lived 
experiences and communicative processes. Qualitative analyses provide insights into the par- 
ticipants’ subjective experiences and meanings they give to their social experiences. Addi- 
tionally, especially pertaining to Communication, qualitative methods allow for participants 
to share their experiences, thoughts, feelings, and perspectives using their own words and 
phrases. In contrast, quantitative instruments such as questionnaires typically have pre-deter- 
mined categories and response options that are focused on generalizability and summarizing 
the data using numbers. 

Qualitative approaches help researchers understand and unpack the meanings that are 
ascribed to texts, artifacts, activities, situations, and experiences. Qualitative approaches often 
focus on describing and interpreting the subjective reality using “thick” descriptive data within 
specific cultural contexts. Thick description refers to close reading and detailed description 
of the experience, including specific contextualized information to help the reader under- 
stand the phenomenon being examined. 

In qualitative research, there is an emphasis on human subjectivity and reflexivity. Reflexiv- 
ity refers to reflections on the role of the researcher’s own biases and viewpoints in shaping the 
data analysis and interpretations (Creswell, 2014). Unlike quantitative methods that emphasize 
objectivity, there is a greater sensitivity to the subjective positionality of the researcher in quali- 
tative methods. Researchers should pay attention to how their own role in the research project 
might potentially influence and shape their interpretations. They not only study language and 
communication that is expressed but also incorporate their subjective interpretations of their 
observations by using their tacit knowledge such as intuition, felt emotions, and latent mean- 
ings. They look for existing patterns, significant symbols, and meanings. The researcher’s own 
biases, perceptions, positionality, and worldviews shape key decisions in the research process. 

Related to this aspect of qualitative research is embodied practice, which means that 
researchers use their own selves as research instruments rather than a survey questionnaire 
or dependent measures in an experiment (Schutt, 2015). They place their bodies within 
the context that they are studying. Even if researchers are not actively participating within a 
specific community for their research but instead are using case studies or analyzing textual, 
visual, audio, or archival documents of various kinds, the emphasis is on bringing in the sub- 
jective perceptions and insights of the researchers. Often this process involves the researcher 
immersing themselves into the context that they are studying for long periods of time. 
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In the next section of the chapter, we will review some of the most popular qualitative 
research methods used in Communication: textual analysis, ethnography, in-depth interview- 
ing, focus group discussions, the case study approach, and arts-based research. There are many 
nuances within each of these methods, which are not covered here in depth. Given that the 
focus of this book is on quantitative methods, we will only provide a brief overview of a 
few qualitative methods here from a social science and critical perspective. There are other 
qualitative research methods—such as life history, contemplative research methods focused on 
self-inquiry and holism, and critical geomapping, which are also emerging within Commu- 
nication more recently—that we are not able to cover here in this chapter, given space limi- 
tations. It is best to consult with other books devoted more in-depth to qualitative research 
methods to learn about the philosophy, history, design, ethics, data analysis, implementation, 
and dissemination of qualitative research. 


Textual Analysis 


Textual analysis is a useful qualitative method that was developed within Communication, 
Media Studies, and Cultural Studies to analyze and interpret texts within the cultural contexts 
in which they are produced and consumed. Textual analysis is an umbrella term that encom- 
passes other types of qualitative approaches such as narrative analysis, discourse analysis, the- 
matic analysis, rhetorical criticism, archival analysis, performance studies, interaction analysis, 
visual rhetoric, and so on. It is a nonreactive or non-interactive method because the texts 
tend to be nonliving data. Texts could be audio, visual, or multimodal audiovisual data. They 
could include a wide range of artifacts such as transcripts, social media posts, news articles, 
TV shows, films, podcasts, photographs, videos, advertisements, graffiti, murals, clothing, and 
so on. 

Whereas quantitative content analysis focuses on categorizing, enumerating, and establish- 
ing relationships among the patterns observed in texts, qualitative textual analysis focuses on 
understanding how specific texts are used within specific cultures, contexts, peoples, places, 
and times. Beyond the texts themselves, the historical, socio-cultural, and political context in 
which the text is embedded is also examined in textual analysis. For instance, analyzing how 
gender identities are constructed through online discourses of motherhood during COVID- 
19, how immigrants are Othered in medical contexts, or examining hip-hop music videos 
through feminist lenses are some examples of how textual analysis could be used within social 
justice scholarship in Communication. 

Researchers immerse themselves into the text to gain interpretive insights and make sense 
of them within specific cultural contexts. Grounded theory is an inductive approach to data 
analysis, which is commonly used in qualitative textual analysis to discover patterns through 
constant comparisons. Using a constant comparison approach typically means identifying 
themes in qualitative data by comparing each observation against all other observations to 
find patterns of similarities and differences. This iterative approach offers flexibility in theory- 
building, which is especially useful for beginning researchers. Computer-assisted coding 
software such as NVivo, ATLAS.ti, MAXQDA, NUDX*IST, and Dedoose can help to save 
time and help to organize data. Several of these software support data visualization, multilin- 
gual analysis, annotations, and word mapping. However, this software can only aid in organ- 
izing the data. Ultimately, the analysis is to be done through subjective reading of the data by 
the researchers. 
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Figure 10.3 Organizing notes for a textual analysis project 


Source: Photo by Tai Jyun Chang on Unsplash, https://unsplash.com/photos/vJfPJfq24S0 


Ethnography 


Ethnography is a qualitative method that involves the study of cultures through participant 
observation, typically over long periods of time. Participant observation is a commonly 
used qualitative method for data collection in which a researcher is both the observer of and 
participant in the culture that they are examining. Participant observation allows researchers to 
understand the nuances and complexities of a culture through observing people within their 
natural settings and gain first-hand experience by immersing themselves into the everyday 
activities of that culture. It emphasizes naturalistic inquiry or field research. Rooted in the 
ideas of phenomenology and naturalism, naturalistic inquiry involves observing and analyz- 
ing research participants in their everyday situations (Schutt, 2015). Whereas field research and 
naturalistic inquiry are broader terms, ethnography is specific to the study of cultures. Applying 
it to Communication scholarship, ethnographers study communication in its natural environ- 
ment in terms of how people interact with one another and other artifacts within the cul- 
ture. Ethnographers are interested in studying the meanings that people give to behaviors of 
themselves and those in their community. They examine the everyday, common, and taken- 
for-granted aspects of communication without any preconceived hypotheses or expectations. 
Ethnographers uncover latent meanings and symbol systems within the culture while still 
preserving the integrity of the qualities of the naturally occurring phenomena. These could 
include rituals, habits, routines, and patterns of language, nonverbal communication, or other 
behaviors of interest to the researchers. 
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Qualitative researchers should be aware of their insider-outsider status. For instance, 
even if a researcher might see themselves as an insider within the community they are 
studying, it is possible that members of the community see them more as an outsider. 
Although researchers might share some characteristics with the marginalized groups, there 
could be other aspects that they do not share, which influence insider-outsider perceptions. 
It is important for researchers, especially those who are committed to social justice research, 
to take the differences in power, status, and positionality into consideration in the research 
process. 

While conducting ethnography, researchers have to decide the extent to which they will 
function as a complete observer or a complete participant in the natural setting that they are 
observing. A covert observer is one who does not participate within the culture but simply 
observes the group. They are publicly identified as a researcher. In contrast, the covert par- 
ticipant is a researcher who fully immerses themselves into the setting and the activities of 
the culture without revealing their research identity. As a participant observer, the researcher 
might start as a full participant and then reveal their research agenda later on because their 
presence as a researcher might influence how participants behave in the setting. Between these 
two positions there can be any number of other variations. Also, it is possible that through the 
course of a research study, the roles that the researcher plays change as the research evolves. 
Depending on the context, the researcher’s positionality, the goals of the project, the socio- 
cultural and political contexts, and ethical aspects of the research, the researcher has to decide 
to what extent they would be a participant versus an observer. 

Within Communication, media ethnography—and more recently, digital or cyber-eth- 
nography—is a newer field of study where the role of media in everyday routines, practices, 
and rituals is observed. It involves observing mediated cultures, including digital and virtual 
spaces, to understand and account for how media users inhabit these spaces and make sense 
of their interactions with others. While ethnographers might not be physically present within 
the culture that they are studying in cyberspace, they are still embedded within the research 
context through digitally embodied practice. For instance, within social justice-related topics, 
this could involve conducting a digital ethnography of gamers committed to social change, 
studying hate groups online, or analyzing YouTube or social media network communities 
online. 

Another specific type of ethnographic method is called autoethnography. Here the 
researcher uses personal, autobiographical, self-data to analyze and interpret the world 
around them. They connect their personal experiences to the larger socio-cultural context. 
In other words, the rigorous inquiry into their own experience becomes the starting point 
to understanding a larger issue. For example, if the researcher is interested in social justice 
such as health and wellness, anti-racism, sexual violence prevention, or reproductive justice, 
they would delve into their own experiences with the topic as a way to then systematically 
interpret and critique the culture, connect back with existing research, and be aware of the 
audience to have an impact on them. The researcher typically uses daily journaling and 
other memos of their experiences but could also incorporate other forms of fieldwork such 
as in-depth interviewing and textual analysis. While embodied practice is a part of almost 
all types of qualitative research, it is especially so with autoethnography, which requires vul- 
nerability in digging deeper into one’s experiences as well as sharing them with a broader 
audience, which could be an emotional experience. Therefore, researchers should make sure 
they are comfortable doing so and incorporate self-care practices while engaging in this type 
of method. 
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Figure 10.4 Taking notes and journaling during an ethnography study 
Source: Photo by Eye for Ebony on Unsplash, https://unsplash.com/photos/7EpZ2IMxK7g 


In-Depth Interviewing 


In-depth interviewing is another common qualitative research method used within Com- 
munication research, including in mixed methods projects. It uses conversation between the 
interviewer and the interviewee as a research tool to draw on more informal storytelling and 
narrative approaches from daily life as compared to more structured survey designs. Interviews 
go deep into an individual’s attitudes, beliefs, opinions, feelings, and experiences. 

Unlike surveys that are more structured with close-ended options, in-depth interviewing 
is designed to allow participants to use their own language to provide details of meanings 
they ascribe to their situations and experiences. Qualitative interviews use a semi-structured 
format. Semi-structured means that although the researcher likely has a predetermined set 
of open-ended questions to pose, those questions are not necessarily asked in a fixed order, 
and there is room for probing certain answers further with follow-up questions as needed. 
The questions could also vary slightly from participant to participant, depending on the con- 
text and their responses. This method allows respondents to frame and respond to questions 
through their own words and experiences rather than the answer options decided a priori by 
the researchers, as is common in surveys and experiments. 

In-depth interviewing allows the researcher to be thorough and gain a comprehensive 
understanding of the participant’s background, experiences, and perspectives and how they 
make sense of them. Active listening is especially important to keep in mind while using 
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in-depth interviews for social justice research. Researchers should approach their interviews 
with humility and a willingness to accommodate the interviewee’s preference as much as 
possible. Especially with social justice related topics, it takes time for interviewees to warm 
up to the researcher and build a rapport with them before discussing issues of injustices and 
inequalities. 

Since several social connections are made through online communications, in-depth 
interviews are increasingly being conducted through phone, email, and video chats. 
Depending on the mode of interviewing, the conversations could be audio or videotaped 
for further transcription and analysis. However, some nonverbal communication informa- 
tion could get lost while using phone or email based interviewing rather than face-to-face 
or video chats. 


Focus Group Discussions 


Another type of interview technique is focus group discussion, which involves in-depth 
interviewing of several participants at the same time and in the same location. Most focus 
groups have about five to 12 participants. They typically meet from one to two hours to 
discuss a particular research topic in depth. In a group setting, people might feel more com- 
fortable to share their perspectives freely. Focus group discussions allow for participants to 
listen to and build on the comments and perspectives of other participants. The discussions 
are facilitated by trained moderators and occasionally also include additional observers and 
note takers. 


Figure 10.5 Focus groups provide rich qualitative data 


Source: Creative Commons/ Wikipedia (https://commons.wikimedia.org/wiki/File:Christ_University_focus_group_6.jpg) 
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The design of questions for focus group discussions often follows the funnel structure 
such that broad topics that are less sensitive are discussed first before moving to more focused 
and specific issues. Just like with in-depth interviewing, the interview guide is semi-struc- 
tured and quite fluid in format. Focus groups provide rich data with thick descriptions using 
the participants’ own language, experiences, and viewpoints. 

Typically, focus groups are homogenous groups. While conducting research on social jus- 
tice topics, in particular, it is important to pay attention to group composition and to be 
sensitive to power relations and cultural differences within groups, which might hinder honest 
and smooth conversations. For instance, if a researcher conducts focus groups to understand 
how diversity initiatives are received by a campus community, it would be best to separate 
faculty groups from staff groups and student groups because of the power differences between 
these groups. Moderators should make everyone feel included and comfortable, drawing in 
participants who might be shy while making sure that dominant participants give more space 
for others to participate. 


Case Study Approach 


Case study is not a single method per se but is one way of approaching qualitative data analysis 
by examining a particular case as a whole, rather than just discrete elements. Here the “case” 
that is studied could be a particular organization, community, social group, family, cause, or 
even an individual person. The “reality” of the case is not divided up into independent or 
dependent variables but rather is examined more holistically as a slice of life. The case study 
approach typically involves a thick description of the situation or problem examined to get a 
sense of the standpoints of various stakeholders involved in a naturalistic setting. Case stud- 
ies have especially been used in subfields such as organizational communication, marketing 
communication, and health communication (Ray, 2005). For example, case studies could be 
used to bring to light specific scenarios relating to a particular health condition, examine how 
a nonprofit organization dealt with crisis situations, or consider ethical dilemmas within a 
public relations case. Case studies have served as useful pedagogical tools in the classroom to 
connect theory to practice using vivid and relevant information to illustrate otherwise abstract 
concepts and theories. 


Arts-Based Research 


More recently, Communication as a discipline has started incorporating social sciences with 
fine arts to use aesthetics, creativity, and artistic skills to describe, evoke, and engage with 
research topics as well as to disrupt, question, and challenge the status quo through a critical 
perspective. Researchers have used such methods as poetry, photovoice, monologues, fiction, 
literature, and visual arts to interpret data in ways that could engage participants in meaningful 
ways about social issues that they care about. For instance, in the Bates and colleagues project 
(2019) in Ecuador, the researchers first conducted in-depth interviews, which helped guide 
an arts-based community project relating to healthy lifestyles. Children of the community 
worked on taking the discussions and dialogues around healthy living and happiness to cre- 
ate their artwork. They were provided with markers, crayons, colored pencils, and paper to 
explore ideas about their shared future/dreams as a community, ways to preserve their com- 
munity culture, and on the theme of healthy living. Once the rough sketch was finalized, the 
artwork was created on a larger scale as a mural wall painting, which was then shared with the 
community for further feedback and dialogue. 
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Figure 10.6 One of the mural wall paintings made by Ecuadorian children in the arts-based qualitative 
methods project on healthy living by Bates and colleagues project (2019) 


Source: Bates and colleagues project (2019) 


Research in Action 


The Center for Scholars and Storytellers 


The Center for Scholars and Storytellers was founded in 2018 by Dr. Yalda T. Uhls, a 
developmental psychologist from University of California at Los Angeles who has also 
worked in the media industry as a film executive with leading organizations such as 
Sony and MGM. This center aims to connect scholars and storytellers committed to 
youth engagement and development. Based in Los Angeles, it draws on the expertise of 
people from both academia and the entertainment industry to bring about transforma- 
tive changes to children and young people aged 2 to 20 years. 

Mixed methods approaches such as surveys, interventions, focus group discussions, 
and content analysis have been routinely used by the Center for Scholars and Story- 
telling. The studies are typically funded using grants from nonprofits and foundations, 
especially those that focus on youth development. In this Research in Action spotlight, 
we showcase one such research project led by Dr. Uhls and team so that readers can see 
how mixed methods research is used to tackle communication and media issues relating 
to positive youth development. 

This mixed methods research study was based on an initiative started by a nonprofit 
organization called Common Sense Media and was funded by John Templeton Foun- 
dation and the Bezos Family Foundation. The research report is titled “The Power 
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of Storytelling: Media and Positive Character Development” and was conducted by 
researchers Andrew J. Sanders, Laurel Felt, Katherine Wong, and Yalda T. Uhls (2019). 

The research project aims at providing useful resources to parents and teachers of 
young people to use the power of storytelling and media narratives to nurture positive 
traits and characteristics in youth. Common Sense Media has created a ratings and tag- 
ging system to identify media content that promotes positive youth portrayals relating to 
character building, role modeling, and other positive characteristics. This project studies 
the effectiveness of this initiative using the following methods: 


e Focus group discussions 
e Pre- and post-exposure online surveys 
e Media intervention study 


The focus group discussions were conducted in English and Spanish with 24 parents 
of kids aged 4 to14 years across three cities. They helped the researchers understand 
parents’ decision-making processes regarding media selection, their perceptions of how 
media content influences character formation in children, and their feedback about the 
characters selected through the Common Sense Media ratings and tagging system. 

The feedback from the focus group discussions helped the researchers design an 
online survey that was administered at two points of time—pre-exposure and post- 
exposure to their online modules. The survey was designed like a self-guided online 
training module about the character content created by Common Sense Media. The 
questionnaire includes items about the media use of families and parents’ attitudes about 
media and character development. 

Finally, the survey results enabled the researchers to conduct a small media interven- 
tion study. This was similar to a field experiment in a naturalistic setting where parents 
and children were asked to view age-appropriate television programs that centered on 
positive character content. Following this television intervention, the parents and chil- 
dren were allowed to either have a free-flowing discussion without any guidance from 
the researchers or used a much more structured approach with guided discussions. The 
researchers analyzed the amount of discussion and also the types of topics discussed by 
parents and children in both these treatment conditions. 

Through this research project, the researchers were especially interested in media lit- 
eracy, how parents and educators could harness the power of media for social good, and 
work toward policies that could encourage more positive portrayals of youth in enter- 
tainment media so that media could serve as a tool for character development and indi- 
vidual transformation, leading to social change at the family and then community levels. 


Types of Mixed Methods Approaches 


There are a number of ways in which quantitative and qualitative methods can be combined 
within a project. Both qualitative and quantitative methods might be given equal importance 
in the project, or it could be that one is given more priority than the other. They might 
be conducted at the same time or one after the other. Indeed, mixed methods designs can 
be broadly divided into two types: sequential designs and concurrent designs, based on the 
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sequence and temporal order of the methods used within the project. Sequential designs 
are when one method precedes another such that one method is used to elaborate upon or 
explain the findings from another method within the same research project. 

There are two types of sequential designs (Figure 10.7). One is QUAL—>QUAN where 
qualitative research precedes quantitative research. For instance, questions that will end up 
in a survey can first be refined and fine-tuned through focus group discussions or in-depth 
interviews. Or a textual analysis might be conducted on stimulus materials that are then used 
for an experiment. Another type of sequential design is QUAN—>QUAL where quantitative 
methods are conducted first before qualitative methods. In-depth interviews could be used 
to understand in greater depth the results from a survey that was previously conducted, for 
instance. Or a survey might reveal specific case studies that need to be examined more in 
depth as examples or as anomalies. 

In contrast to sequential designs, there are concurrent designs where the qualitative and 
quantitative methods happen simultaneously or concurrently rather than one after another. 
Concurrent designs can be represented visually as QUAL + QUAN (note the + sign here 
instead of the — sign in the previous paragraph). 

Let us take the example of an excellent social justice-oriented mixed methods project by 
the two authors— Sarah Jackson and Brooke Foucault Welles (2015)—who are featured later 
in this chapter in the Scholar Spotlight section. In their project on hashtag activism, they 
examined the #myNYPD hashtag, which was used by activists to highlight police brutality 
and racial profiling by the New York City Police Department (NYPD). Apart from analyzing 
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Figure 10.7 Types of mixed methods research 


Source: Emily Riewestahl 
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Twitter data, the researchers were also interested in the mainstream news media coverage of 
this hashtag. In the first stage of the research project, the authors used a quantitative method 
called social network analysis. The social network analysis used mathematical modeling of 
tweets based on mentions and retweets to map out the ways that tweets clustered meaning- 
fully together around the most popular sources of information on Twitter for this hashtag. 
In the next stage of the project, a sample of tweets from these key sources identified from 
the network analysis were analyzed using qualitative discourse analysis to make sense of how 
language is used to create meaning and conversations around social issues. In the final stage of 
the project, the researchers analyzed mainstream news outlets using textual analysis to exam- 
ine how these news sources covered and framed this hashtag similarly or differently from the 
online media campaign on Twitter. 

Another excellent example of mixed methods research on social justice issues is by Satveer 
Kaur-Gill and colleagues (2019) in which the researchers conduct a media framing analysis of 
representations of Singaporean foreign domestic workers. The discrimination against migrant 
domestic workers is often ignored by mainstream news discourses. Stemming from a human 
rights framework that takes into account everyday violence against those at the intersections 
of gender, domestic work, and transnational migration in Southeast Asian contexts, this study 
analyzes media discourses in the newspaper, the Straits Times. In the first phase a grounded 
theory approach was taken to conduct open coding of news content using an inductive 
method. Following this step, in the final stage, a quantitative content analysis was conducted 
to examine the patterns and clusters in the data. 

Some designs are much more complex, combining both sequential and concurrent designs. 
While such studies have been conducted, they are quite rare and are resource intensive, poten- 
tially taking up several years and many researchers in the team to complete. For example, in 
the context of understanding public discourses on genetics, Condit and colleagues (2002) first 
conducted an audience analysis and then a news analysis in the second stage. Within the first 
stage of the project involving audience analysis, a quantitative survey of 122 undergraduate 
students was used concurrently with a qualitative study with a general non-student popu- 
lation. The survey examined audiences’ perceptions about genetics. The qualitative study 
involved in-depth interviews with 95 participants to discuss what came to their minds when 
they thought about various metaphors associated with genes. Once this stage of the research 
project was completed, the researchers conducted a rhetorical analysis of newspaper articles 
published over a two-year period to understand the ways in which the media framed these 
issues. Finally, they conducted 16 focus group discussions to respond to some of these texts 
that were analyzed. 

Another way to divide mixed methods designs is based on priority and importance. Within 
sequential designs, when one method is dominant and given priority over another, it is called 
a staged design. For instance, a sequential design where quantitative methods are prioritized 
over qualitative methods would be represented this way: QUAN—qual (note that QUAN 
is represented with capital letters to indicate that it was the primary method). Here the main 
and first method is quantitative, whereas the secondary method is qualitative. A staged design 
that prioritizes qualitative methods would look like this: QUAL— quan. Concurrent designs 
could also prioritize one method over another. They are referred to as embedded designs 
(QUAN + qual or QUAL + quan). For example, when within a focus group discussion par- 
ticipants complete a brief survey questionnaire, it would be an embedded design where quali- 
tative methods are the primary methods within which quantitative methods are embedded. 
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For instance, Katharine Head and colleagues (2019), used a QUAN + qual study design 
to examine how cervical cancer communication and screening happens at health cent- 
ers. Their study was predominantly a quantitative survey. However, they included some 
open-ended questions on the survey questionnaire, which were later analyzed qualitatively 
using textual analysis. Both the quantitative and qualitative methods were administered 
concurrently rather than sequentially. In contrast, Daniel Park and colleagues (2017) used a 
sequential QUAN + QUAL design where both survey data and focus groups were equally 
important to the project. This was part of a feasibility study for a new mobile app called 
Smartphone Medication Reminder App (SMRA) to determine whether this technology 
would work as an effective intervention for patients to remember to take their medication, 
especially within middle-aged and older patient populations with chronic illnesses. After a 
hands-on interactive session using the app, participants completed a survey questionnaire 
about their experiences. Those participants who received training to use the app were 
compared with those with no training. Following the survey, focus group discussions were 
conducted using semi-structured interviews to understand the challenges, struggles, and 
perceptions of participants. 


Scholar Spotlight: Dr. Sarah Jackson and Dr. Brooke Foucault 
Welles 


Sarah J. Jackson is Presidential Associate Professor at the Annenberg School for 
Communication at the University of Pennsylvania. A scholar of the public sphere, 
Sarah J. Jackson’s expertise lies at the intersection of media and activism, with a focus 
on the ways Black and feminist activists make media and, in turn, how these groups 
are represented in media. She’s a founding member of the Center for Critical Race 
and Digital Studies and board member of MediaWell at the Social Science Research 
Council. 

Brooke Foucault Welles is an associate professor in the Department of Com- 
munication Studies and a core faculty member of the Network Science Institute at 
Northeastern University. Combining the methods of computational social science and 
network science with the theories of communication studies, Foucault Welles studies 
how online communication networks enable and constrain behavior, with particular 
emphasis on how these networks enable the pursuit of individual, team, and collective 
goals. 


Q: What are some social justice issues that motivate you as Communication researchers? 

Dr. Jackson: 1 chose to become a communication scholar because of deep 
concerns about whose voices are and are not included in our democracy 
and the radical power of the stories told from the margins. In my work 
this has meant I primarily study African American and feminist media and 
activism. 
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Figure 10.8a Scholar Spotlight: Dr. Sarah Jackson 


Dr. Foucault Welles: I started my research career around the same time that online 
communication became widely available in the U.S. From early days of usenet 
groups, through I.R.C., to the social media of today, I am fascinated by the 
capacity of online communication to bring together and amplify marginalized 
voices while at the same time reproducing many of the systems of power and 
oppression that keep people from thriving. Finding ways to harness the social 
justice possibilities of the internet while managing its oppressive potential is 
one of the most profound challenges for communication researchers working 
today. 

Q: When did you first start collaborating? How has your experience been working together? 
What are some strategies that have worked best for you in making your research collabora- 
tions work? 

Dr. Jackson: We first began collaborating in 2014, and it has been a joy and built 
a solid friendship in the process. I think what really made it work for us is that 
we recognized that we each had an important set of knowledge necessary to 
answer questions about the power of counterpublic networks. We usually work 
in a shared document and outline before we jump in; it also helped that for years 
our offices were next door to one another. 
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Figure 10.8b Scholar Spotlight: Dr. Brooke Foucault Welles 


Dr. Foucault Welles: Our work together started with a specific project (our #myN- 
YPD paper) but quickly grew into a trusting friendship. This trust makes it 
easy to work together. We have very different areas of expertise and different 
ways of understanding the world, but I trust that Sarah is truly the best at what 
she does (and I think she would say the same of me). Because our skills are 
very complementary, we can hand off pieces of work we find challenging, and 
the other usually knows how to move things forward. It is also important to 
acknowledge that social justice research can be emotionally challenging. Read- 
ing thousands of tweets about things like police brutality, domestic violence, 
and sexual assault can be heartbreaking, enraging, and exhausting. I cannot 
imagine doing this work without a trusted friend to lean on when it gets 
overwhelming. 

Q: What do you see as the role of quantitative research in furthering social justice in 
Communication? 

Dr. Foucault Welles: We are in a paradigm-shifting moment in Communication and 
social justice research. For generations, we had theories that could not truly be 
empirically tested because we had no reasonable way to gather complex com- 
munication data. Digital communication, and the trace data it leaves behind, 
allows us to gather detailed records of communication across modes, scales, and 
levels of analysis. Quantitative and computational methods make it possible to 
match these data to test long-standing theories of social justice communication, 
as well as develop new theories inspired by patterns of communication we can 
now observe. 
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Q: How about qualitative research? In what ways does qualitative research contribute to social 
justice goals in Communication? 

Dr. Jackson: Qualitative approaches are at the very crux of questions of power and 
culture in our field as Stuart Hall and so many others illustrated long ago. I believe 
that if our scholarship is going to document and aid movements for justice, criti- 
cal qualitative scholars have already shown us the way. My hope is that these 
approaches will become more central to all communication education so that 
quantitative scholars weigh them with the same importance as their own. 

Q: Based on your experiences, what are some limits or challenges of using mixed methods as a 
means of engaging in social justice efforts that researchers should be aware of? 

Dr. Jackson: It takes longer! (As does any form of collaboration in which people have 
to learn to engage across approaches they might be less familiar with.) 

Dr. Foucault Welles: It definitely takes longer, not only to synthesize insights across 
multiple methods, but also to develop a vocabulary that integrates different ways 
of thinking. It is not easy to write papers that include critical insights and math- 
ematical formulas in ways that are accessible to a broad audience of scholars. As 
we developed our theoretical insights, we also had to develop models for talking 
about our work. 

Q: Despite these challenges, what are some benefits of doing mixed methods research, especially 
as it relates to social justice? 

Dr. Jackson: Some questions can be answered more fully with mixed methods research. 
When these questions are related to justice issues, it is absolutely worth it to use all 
the methods available to us to paint a fuller picture, especially when that picture 
will serve those working for social change and document their contributions to 
our world. 

Dr. Foucault Welles: When mixed methods are done well, you reach a lot of different 
constituencies. Some people will never be motivated to act on poignant exam- 
ples without numbers to back them up. Others will never believe the urgency of 
the numbers without compelling examples to back them up. When you bring 
together multiple approaches, you have the potential to convince broad audiences 
of the urgency of the social justice issues you study. 


Mixed Methods for Social Justice in Communication: Some 
Considerations 


As critical researchers committed to using research to dismantle oppressive structures and 
question hierarchies, both quantitative and qualitative methods are helpful to serve the goals of 
social justice, transformation, and equity. Donna Mertens, leading scholar of the transforma- 
tive paradigm, defines mixed methods for social justice as follows: 


Mixed methods research, when undertaken from a transformative stance, is the use of 
qualitative and quantitative methods that allow for the collection of data about historical 
and contextual factors, with special emphasis on issues of power that can influence the 
achievement of social justice and avoidance of oppression. 

(Donna Mertens as cited in Johnson et al., 2007, p. 120) 
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Mixing research methods can be done for a variety of reasons, but they are all primarily 
related to increasing the validity and expanding the knowledge about social issues. Greene 
(2007) lists some of the common purposes of using mixed methods approaches: 


e Discover: to discover new perspectives or frameworks from seemingly paradoxical or 
contradictory information 

e Expand: to extend the range of inquiry to be able to answer different aspects of a prob- 
lem or issue using various methods 

* Converge: to check whether results across different methods correspond and coincide 
with one another 

* Complement: to seek clarification or elaboration of the results of one method with 
those of another method 

* Develop: to use the results of one method to build or inform the development of another 
measure or otherwise inform methodological decisions 


For example, quantitative data collection such as surveys might be followed by qualitative 
methods such as focus group discussions to explain or make sense of the quantitative data. 
Alternatively, a qualitative phase of data collection such as in-depth interviews 1s followed by 
a quantitative method such as a national survey to test the generalizability of the initial quali- 
tative findings. A well-designed and carefully thought-out mixed methods project could be 
the “best of both worlds,” which could benefit from the insights and perspectives gained from 
both qualitative and quantitative research methods. However, it is naïve to assume that just 
because one uses mixed methods research, it automatically overcomes the drawbacks and dis- 
advantages of using any one method. A researcher needs to carefully think about the purpose, 
sequence, and resources needed as they design a mixed methods project. 

With the launching of a new research project using mixed methods, researchers should 
review the literature by searching broadly for prior quantitative, qualitative, and mixed meth- 
ods research within Communication and other allied disciplines. These literature searches 
help researchers to get a good grasp of historical, social, and cultural contexts relating to the 
social issue of interest. Quantitative data such as secondary data on demographic information 
or prior surveys conducted on the topic or the community could be key sources of infor- 
mation that help to understand the issue and define the problem clearly. If the researcher is 
conducting community-based mixed methods research for social justice, it can be helpful 
to involve community members in defining the problems of concern through qualitative 
methods such as in-depth interviews and focus group discussions in the initial stages of the 
research process. For research projects focused on social justice, giving communities agency in 
providing input into the research process can help build trust and rapport to pick the appro- 
priate theoretical frameworks and ask the right research questions in order to successfully 
bring about the transformative change they seek. We will learn more about community-based 
research in the next chapter. 

Qualitative researchers need to be trained well to be sensitive to the context and cultures in 
which they are examining the phenomenon of interest and be able to understand unwritten 
underlying meanings. In picking theoretical frameworks that guide mixed methods projects, 
researchers from academic backgrounds without much contact or connection with the com- 
munities that they are examining should especially invest considerable time into this step of 
the research design process. Doing so can help ensure that their research is not unduly guided 
by their own biases and worldviews but through actively listening to the groups that they 
study. Doing this could be time consuming, and it is a skill that does not come easily to some 
researchers. For instance, if conducting an ethnographic research study, the researcher needs 
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to be able to facilitate the conversation, build a rapport with the participants, and put them 
at ease in their natural environments. Therefore, it often means that qualitative researchers 
not only design the studies but often personally conduct the interviews, focus groups, or 
ethnographic research as part of fieldwork by immersing themselves in the culture for several 
months, if not years. Apart from field notes and transcriptions, data management and analyses 
could also take a long time. Perseverance, time management, and self-care strategies are help- 
ful to avoid burnout and to stay motivated to complete the project. 

With social justice issues, it is likely that the researchers are personally invested heavily in 
the topic and that there is long-term involvement in the project. Because qualitative research- 
ers who do field research become embedded within the contexts and cultures that they are 
studying, they need to consider how such involvement might affect them personally. Nov- 
ice researchers should be aware that there could be moments of emotional conflicts, ethical 
dilemmas, and other unanticipated situations that might come up, especially in doing social 
justice research projects. Immersing themselves into the research context as a participant, for 
instance, could make it hard and awkward at times to step back into the role of the researcher. 
For example, researchers might need to get the permission of gatekeepers in leadership roles 
to access the populations they wish to study. However, doing so could make it hard for the 
researcher to critique the system or question those in such positions of authority. There is no 
set formula for how to manage these relationships as both a researcher and a participant. The 
researcher needs to balance these different roles and draw boundaries where needed to be 
clear on expectations. A lot of the learning comes from everyday practice and depends on the 
individual researcher’s style and the specific culture or topic that they are studying. It would 
be helpful for novice researchers to anticipate such ethical challenges ahead of time and think 
about how they would navigate them. With practice, especially under the guidance of more 
experienced researchers, it is possible to navigate such situations successfully. 

In terms of data collection and analysis, different types of expertise are required for quan- 
titative versus qualitative research. For social justice issues, in particular, it is important for 
researchers to consider the ways in which they can bring in as much diversity as possible 
within the project to make sure that marginalized voices are adequately represented. Mixed 
methods research can be conducted by a single researcher who is well versed in design, 
data collection, and data analyses of qualitative and quantitative research. Typically, though, 
researchers with a set of research expertise—qualitative or quantitative—team up with others 
whose research skills complement their own and thus add to the broader understanding of the 
research problem. Such collaborations between quantitative and qualitative researchers can be 
productive and meaningful, especially if they are motivated by similar problems and social, 
political, or cultural issues. 

Once the data have been collected and analyzed, they should be used to facilitate the 
transformative change on the ground that the researchers seek to bring about. Making the 
research findings available publicly and in an accessible manner is important to get partici- 
pant inputs. Researchers could also consider ways of facilitating the social change process 
that could follow from the research project. For example, researchers could share research 
findings through social media, newspaper articles, a town hall event or blog post to facilitate 
conversations about bringing about social transformation. Public scholarship is one aspect 
of the communication of social justice research where scholarship is shared broadly beyond 
academic audiences through more popular outlets such as opinion pieces, documentary films, 
and newspaper articles. This topic will be dealt with in greater depth in the final chapter of 
this book. 

One of the challenges of mixed methods research is that it requires time, expertise, and 
often sustained funding to complete. These resources might not be readily available to 
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researchers, especially those working on community-oriented social justice work. Some social 
justice issues might be urgently in need of addressing. Most community organizations sim- 
ply do not have the time or money to devote to longitudinal or multi-stage mixed methods 
research projects. Researchers, especially early career ones, may not have time and funding 
to devote to long-term mixed methods projects. Another important consideration relates 
to ethical challenges that might arise from using complex designs in both concurrent and 
sequential designs where participants’ consent needs to be negotiated at every stage for differ- 
ent methods, which might lead to sample attrition. In terms of data analysis, if the findings 
from qualitative and quantitative aspects of the study contradict one another, they need to be 
addressed and resolved. It could become difficult to compare findings from these two aspects 
of a project or to know how to account for such discrepancies. Such a possibility introduces 
yet another level of complexity that the researcher must manage while conducting mixed 
methods research for social justice. 


Summary 


Using multiple perspectives and viewpoints to approach a research problem is called triangula- 
tion. Mixed methods research allows for methodological triangulation by using both quanti- 
tative and qualitative methods to address social issues. Using mixed methods approaches can 
help to increase researchers’ confidence in the validity of their findings, resolve or confront 
any contradictions and paradoxes in the data, and, from the point of view of the quantitative 
researcher, help bring in thick descriptive data to complement numerical data from quan- 
titative methods. Qualitative methods are more inductive, flexible, reflexive, and subjective 
in their approach as compared to quantitative research methods. They are characterized by 
naturalistic inquiry, emergent design, embodied practice, and reflexivity. Naturalistic inquiry 
is studying communication behaviors and processes within the natural environments in which 
they occur. Emergent design is where the study hypotheses are not predetermined but evolve 
as the project unfolds. Embodied practice is the qualitative researcher embedding themselves 
as a research instrument within the subculture that they study. Reflexivity is the researcher 
being self-aware and self-reflective about how their worldviews, positionality, values, and 
biases might shape their research decisions. 

Some qualitative methods such as textual analysis, ethnography, in-depth interviewing, 
focus group discussions, case studies, and arts-based research approaches are described briefly 
in this chapter. Textual analysis is a nonreactive qualitative method of subjective reading 
of the meanings of texts to gain insights within specific socio-cultural contexts. It often 
uses grounded theory as an inductive approach to analyze patterns in the data through con- 
stant comparisons. Ethnography is a qualitative method that emphasizes naturalistic inquiry 
through participant observations in their everyday environments to make sense of meanings 
that people give to behaviors of themselves and others in their community. It analyzes rituals, 
habits, routines, and patterns of communication and other behaviors. In-depth interviewing 
uses semi-structured conversational formats to draw on informal storytelling and narrative 
practices. Focus group discussions involve group conversations that allow participants to listen 
to and build on others’ perspectives. A case study approach examines a particular organization, 
community, social group, or individual in detail through thick description and from multiple 
perspectives of stakeholders within that group. Arts-based research uses art techniques such 
as painting, photography, visual arts, and poetry to disrupt, question, and evoke feelings and 
thoughts, often with the intent of challenging status quo norms. These qualitative methods 
can overlap and be used in conjunction with one another as well as with other quantitative 
methods such as surveys, experiments, and content analyses discussed in previous chapters. 


Mixed Methods 299 


Mixing quantitative methods with qualitative methods can be done in multiple ways 
based on which method is dominant in the project as well as the sequencing of the meth- 
ods. Sequential designs are when one method precedes another, often in order to elaborate, 
explain, or build on the previous method. Concurrent designs, in contrast, are when both 
the quantitative and qualitative methods occur simultaneously. Staged designs are those where 
one method (quantitative or qualitative) is more dominant or given greater priority than the 
other method. Embedded designs are a type of concurrent design where one method is given 
priority over the other, which is embedded within the dominant method. 

Although mixed methods can be helpful in triangulation of methods to get a better grasp 
of social justice issues, they can also take a long time to conduct, require a lot of resources, 
and rely on the researcher’s ability to understand, interpret, analyze, and integrate both 
qualitative and quantitative data. Nevertheless, mixed methods approaches can be incredibly 
valuable to critical social justice researchers to answer research questions with greater depth 
and breadth. 


Discussion Questions and Activities 


1. What is triangulation? Explain the different types of triangulation using theories and 
methods that would be relevant to the social, political, or cultural issues of interest to you. 

2. How are qualitative research methods different from quantitative methods? What are the 
advantages and disadvantages of each of these approaches? 

3. If you were to design a qualitative research study from a critical paradigm versus from 
an interpretive/constructivist paradigm, how would your research questions and choices 
differ? 

4. Design a mixed methods study for a social justice topic using qualitative and quantitative 
methods sequentially. Explain the rationale for the sequencing. How will each stage of 
your project inform the next one? 

5. Mixed methods research is certainly beneficial to address research questions around social 
justice issues. What are some challenges that a researcher might encounter while con- 
ducting mixed methods research? How might researchers overcome these challenges and 
anticipated problems? 
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11 Community Partnerships and 
Participatory Research 


In this book, we have presented the steps involved in designing and carrying out quantitative 
research as a series of factors the researcher must consider amid choices the researcher must 
make. What topic are you interested in? What method will you choose? How will you meas- 
ure your concepts or access a research sample? What statistics will you run to understand your 
data? These are, indeed, decisions that researchers must make. 

However, it is important to know that researchers do not always make these decisions 
alone, nor do they make them in a vacuum, devoid of context. In fact, when researchers make 
decisions about how to conduct their research they always do so within a particular context. 
At any point in time in history, communities have their own concerns, experiences, assets, 
and needs. To be attuned to and responsive to these concerns, experiences, assets, and needs, 
researchers can form partnerships that involve community members as active participants in 
research. Teaming up with others in the community has profound potential for understanding 
and intervening in oppression, injustice, and inequalities at the grass-roots level. 

Social justice research often involves partnerships, participation, and collaborations 
between researchers and community members. Such partnerships are especially necessary 
if the researcher is interested in bringing about transformative, actionable change within 
communities. Such research is called community-based research or participatory research, 
although there are minor differences between these terms, which will be discussed later 
in this chapter. Community-based research can incorporate any or all of the methods we 
have described in the previous chapters. This is why we have placed this chapter toward 
the end of the book, after you have had the opportunity to learn about various types 
of quantitative, qualitative, and mixed methods research. Methods such as surveys, field 
experiments, and focus group discussions are especially popular in community-based social 
justice research. 

We can trace back the roots of quantitative community-based research to the 1940s and 
1950s when two women social scientists, Margot Wormser and Claire Selltiz, pioneered com- 
munity self-surveys or audits (Torre & Fine, 2011). Their research helped to provide guid- 
ance on how to conduct social justice research about civil rights on topics such as housing, 
education, and health services. Their inclusive participatory methods focused on reaching as 
many community members as possible by considering as many ways as possible for including, 
involving, and encouraging marginalized groups to participate in the research process. Their 
studies were not just precursors for contemporary critical community-based participatory 
action research but also some early examples of social psychological research for social justice. 
Today, community-based research is especially popular in health-related disciplines such as 
public health, health disparities, nursing, health education, and prevention studies, though it 
has been applied in many other contexts, settings, and topics. 
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The current chapter will delve into some common terminology, principles, and practi- 
cal suggestions to consider when designing, implementing, and assessing community-based 
research projects. Community-based social justice research emphasizes principles such as col- 
laboration, participation, experiential knowledge, action-orientation, and social transforma- 
tion, which will be elaborated on in this chapter. You will also learn about various types of 
participatory research and approaches such as the socio-ecological model, community-based 
participatory research, participatory action research, trauma-informed participatory approach, 
culture-centered approach, program intervention studies, engaged scholarship, and public 
scholarship. The nuts and bolts of planning and implementing community research will be 
discussed, from problem definition to evaluation and feedback. We will end the chapter with 
some practical considerations while conducting such research, such as long-term involve- 
ment, project management, and community consent. 


Types of Communities 


Before we get into community-based research, let us first look at the word community. 
Community is defined as a group of people who share some common characteristics such 
as living in the same geographic space (a particular neighborhood, campus, village, town, 
or state) or with similar interests or identities (a network for Black journalists, a community 
for indigenous media makers, a community choir for low-income immigrants, or a center 
for lesbian couples’ counseling). Communities can include residents of a local area, industry 
professionals, government agencies, nonprofits, practitioners, and any groups of people or 
organizations that are affected by research results. 

In today’s mediated contexts, the word community extends well beyond a specific physical 
space to include virtual communities. Online communities where individuals make new 
connections, network with others, and interact about any number of topics through social 
media networks, mobile apps, video games, chatrooms, video chats, and so on are important 
to understand and pay attention to in an increasingly mediated world. Some considerations 
for community-based participatory research that engage with online communities include 
issues such as barriers to access to technology, offline versus online engagement, media liter- 
acy skills, concerns about privacy, lack of clarity about online content ownership, and lack of 
trust in online spaces. However, the internet and mobile phone technology in particular offer 
excellent opportunities to enhance collaboration and participation, access and share infor- 
mation, reach beyond a limited geographic area, mobilize swiftly, and reduce research costs, 
which are all important considerations in doing community-based participatory research (see 
Tami-Maury, Brown, Lapham, & Chang, 2017). 

Typically, community is a term used to describe participants in the research process who 
are outside of higher education contexts. Often such research discusses university-community 
partnerships where a specific center or institute within higher education reaches out to one 
or two specific community-based organizations to collaborate. For instance, the first author 
of this book has engaged in a long-term community partnership with schools in Massachu- 
setts in implementing media literacy programs, especially relating to social justice issues such 
as gender equality and violence prevention. The second author of this book has conducted 
community-based research on media literacy through her initiative called Media Rise (www. 
mediarisenow.org), which brings together researchers, artists, activists, K-12 teachers, high 
school students, parents, and community leaders to promote meaningful media. However, 
community-based research can also include the campus community or those within research 
organizations. Examples of this might include running a health communication campaign for 
students and faculty within a university or engaging students in racial dialogues about campus. 
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In fact, the second author of this book has founded the Difficult Dialogues Project (www. 
difficultdialoguesproject.org), which is a campus initiative that uses facilitated dialogues to dis- 
cuss difficult topics such as racism and COVID-19 inequalities to build empathy, active listen- 
ing, and bring about systemic change within higher education (Ramasubramanian, Sousa, & 
Gonlin, 2017). 


Community-Based Research in Communication 


Community-based research within Communication is particularly popular in some subfields 
such as organizational communication, health communication, and media literacy. It draws on 
methods such as engaged scholarship, program evaluation, and intervention research (which 
we will elaborate on in the next section of this chapter). For example, organizational commu- 
nication research often involves conducting research within organizational settings outside of 
higher education contexts. Similarly, health communication researchers conduct community- 
based programs that often use intervention research or program evaluation approaches that 
stem from field experiments. 

Each community-based research project is highly specific and tailored to a particular com- 
munity, context, and problem. Most of the community-based research within Communi- 
cation has been related to health communication and has used mixed methods research to 
evaluate health-related programs and interventions (see Obregon & Waisbord, 2012 for many 
examples around the globe). However, community-based research methods have also been 
brought to bear on other social justice issues such as gender justice, environmental issues, 
media literacy, immigrant rights, technology use, and political participation. One exam- 
ple is a study by Kate Magsamen-Conrad and colleagues (2015) that used a pretest/posttest 
experimental design as part of the Intergroup Communication Intervention program to test 
whether technology literacy among older adults in the community was influenced by four 
weeks of one-hour training workshops. Another example of community-based research is a 
project conducted by Warshel (2012) on martyrdom portrayals on television with over 400 
Palestinians over a 2.5 year period. Warshel’s work used a home-based survey of children’s 
television use, multi-sited ethnography of family and media use, critical viewing analyses of 
television texts, and survey research with parents about their children’s television viewing. 

Community-based research is the primary focus of research centers such as the Center for 
Culture-Centered Approach to Research and Evaluation (CARE) directed by Mohan Dutta 
at Massey University in New Zealand. CARE researchers have studied a wide range of topics 
such as food insecurity among tribal communities in India, domestic worker rights, transgen- 
der sex workers’ rights, and other health-related topics in Singapore, as well as topics relating 
to indigenous health, health and migration, and poverty in New Zealand and elsewhere. 
Other Communication scholars around the world have used community-based approaches to 
examine a variety of social justice topics such as stigma around substance abuse among street 
youth in Bogota in Colombia (Ritterbusch, 2016), transgender equality issues in the U.S. 
(Billard, 2019), post-Katrina racism (Lykes & Scheib, 2016), technology enhanced learning 
among youth in Australia and United Kingdom (Marshalsey & Sclater, 2019), technology 
use among older adults in Romania (Schiau, Ivan, & Bird, 2018), and critical media literacy 
among migrant youth in Netherlands (Leurs, Omerović, Bruinenber, & Sprenger, 2018). 


Socio-Ecological Approach to Social Justice Research 


In order to bring about community-level changes, social justice researchers need to consider 
the relationship among individuals, organizations, communities, and societies. Drawing on 
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sociological theories, communities can be seen as lying in between smaller units of change 
such as individuals and larger spheres of influence such as the entire nation. The micro- 
level focuses on changes that need to happen with individuals in interpersonal relationships 
(Figure 11.1). The meso level focuses on organizational change or community-based 
changes. The macro level focuses on changes at the societal or global level. In order to bring 
about social change, it is important to consider all these nested levels of change that influ- 
ence one another. 

These levels of change stem from the socio-ecological model that recognizes that 
there are several circles of influence ranging from the individual level, the interpersonal 
level such as a peer group or family, the organizational level such as workplaces or schools, 
the community level such as a specific neighborhood, and finally the public policy level that 
includes societal change (Figure 11.2). This model is commonly used within health com- 
munication scholarship to consider the multiple levels of ecological contexts to address 
to improve public health. Whereas individual changes can be brought about through 
persuasion, cues to action, and social norms, community-level changes need to address 
more complex processes such as collective decision-making, organizational cultures, poli- 
cymaking, and shared resources. At a broader societal level, social justice and systemic 
changes are influenced by political, legal, cultural, economic, technological, and other 
such macro issues. However, all of these levels are interrelated and intertwined such that 
they influence one another. 


Macro 


(society, community, etc.) 


Meso 


(organizations, schools, etc. 


Levels of Change 


Figure 11.1 Levels of change 


Source: Emily Riewestahl 
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Public Policy 


Socio-Ecological Model 


Figure 11.2 The Socio-Ecological Model 


Source: Emily Riewestahl 


Level of Community Participation 


The level of community involvement can vary tremendously within research collaborations. 
Participation need not always be equal. There will be moments in the research process where 
there might be lesser or more involvement with the community. One can think of such col- 
laboration as a continuum that ranges from researcher-driven projects to community-driven 
projects (Figure 11.3). Projects could be community-oriented, where the research is ori- 
ented toward community goals but does not necessarily involve community members in the 
research process. Community-oriented projects are not typically conducted within commu- 
nity settings. Community-placed research is conducted entirely by researchers but within 
community settings without much involvement of the community. One can say that most 
field experiments and intervention research projects are community-placed projects. The 
level of community involvement is minimal and is limited to occasional consultations, if 
any. Community-based participatory research involves equal partnership where resources, 
credit, and power are shared equally between research investigators and community members. 
Finally, some community-oriented projects are community-driven in the sense that com- 
munity members take the lead in the direction that the research project takes. In the rest of the 
chapter we will use the term community-based research largely to refer to those research projects 
where involvement from communities is quite substantial. 
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Figure 11.3 Level of community participation shown on a continuum from least involvement (left) to most 
involvement (right) of the community 


Source: Emily Riewestahl 


There are many different types of community-based research, which differ slightly from 
one another in origins, emphases, goals, and level of community involvement. They include 
participatory action research, community-based participatory research, program evaluation 
research, engaged scholarship, and public scholarship. These approaches are explained in more 
detail in the sections that follow. 


Participatory Action Research 


Participatory action research (PAR) is a type of community-based research that is directly 
linked to actionable outcomes for the community. It is action-oriented and aims to bring 
about social transformation. The community has far more involvement, participation, 
and influence over the research project than some other approaches discussed in this 
section. It has an explicit emphasis on enabling action through research. It pays careful 
attention to power relations and is deliberate about advocating for shared power between 
academics or other researchers, practitioners, and community members. Communica- 
tion scholars involved in participatory action research have used it to address questions 
about poverty, food insecurity, affordable health care, worker rights, medical education, 
elder care, and climate change in communities by involving community members in the 
co-construction and co-design of research practices such as open forums, online spaces, 
photovoice techniques, geo-mapping, public dialogue sessions, focus group discussions, 
and survey methods. 


Community-Based Participatory Research 


Community-based participatory research (CBPR) is a common approach to commu- 
nity-based research. This can be seen as a type of social justice scholarship that focuses 
on bringing about community change through social transformation. CBPR projects 
tend to be participatory in nature, focused on transformative action, and often use mixed 
methods and multiple theoretical perspectives to reach their social justice goals. This 
means that there is active participation among researchers as well as community organi- 
zations in research design, implementation, and sharing. Community involvement has 
implications for every stage of the research process (Leavy, 2017). Some characteristics 
of CPBR are: 


e Problem-centered approach that actively focuses on solving socially relevant issues 
e Shared decision-making and community involvement in the research process 
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e Iterative approach to data collection and analysis, often using mixed methods (both quan- 
titative and qualitative), which focuses on continuous learning, flexibility, and a focus on 
actionable outcomes 

e Integration of formal and informal sources of information 

e Emphasis on public scholarship that translates research for lay audiences 


Program Evaluation Research 


Apart from approaches that emphasize participation and collaboration, another important 
approach that has shaped community-based research is program evaluation. Program eval- 
uation research emerges from applied social research such as intervention sciences, field 
experiments, policy research, and randomized control trials, especially within health, social 
work, and developmental communication contexts (Linfield & Posavac, 2019). When we 
refer to community-based programs or interventions, they could be any type of organized 
effort toward some shared goal. By intervention in this context, we mean direct action 
toward transformative social change (see Dempsey et al., 2011 for more about commu- 
nity, intervention, and social justice). Given our emphasis on social justice in this book, 
we are focusing on programs or interventions that relate to social justice goals of reducing 
social inequalities through more equitable distribution of shared power and resources for the 
overall well-being and transformation for the community. Some examples are initiatives or 
programs relating to violence prevention, bystander intervention training, health education, 
public health campaigns, media literacy, prejudice reduction, voter registration efforts, and 
civic engagement. 


Engaged Scholarship 


Engaged scholarship has often been used in applied communication and organizational com- 
munication contexts to refer to partnerships and collaborations with industry groups, 
policymakers, government officials, and professionals. Here, the emphasis of engagement 
and community involvement may or may not be oriented toward social justice. Whereas 
engaged scholarship has usually been used to refer to researcher-community partnership 
itself, public scholarship is slightly different. Public scholarship typically refers to the 
ways in which research is communicated to broad public audiences. It focuses on citizen 
outreach, public communication, and translation of scientific information for lay audi- 
ences. Like engaged scholarship, it also incorporates public feedback and inputs in shaping 
research initiatives in a collaborative manner. We will learn more about public scholarship 
in the final chapter. 


Identifying the Topic in Community-Based Research 


The first step in the research process is identifying the topic to study. In terms of selecting 
the research topic relating to social justice and the community to study, there is a wide range 
of issues and topics such as the challenges people face from a human rights (immigrant rights 
or LGBTQ+ rights, for instance) or justice (gender, criminal, economic, or environmen- 
tal justice, for example) perspective. Your research background and specific interests within 
Communication—including, but not limited to organizational communication, media stud- 
ies, health communication, rhetoric, political communication, journalism studies, law and 
policy studies—as well as your personal experiences can shape the type of community-based 
research project you pick to study. 
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Current affairs and topicality also shape the topics and communities that you study. 
For instance, the Difficult Dialogue Project on campus race relations discussed earlier was 
prompted after a racist incident on campus that the second author became aware of through 
the university president’s message to denounce hate and racism. Other factors that could 
influence topic selection are funding opportunities that support specific community-based 
projects. Por instance, several universities have started learning communities for researchers 
interested in community-based scholarship, and many foundations offer large grants to sup- 
port such initiatives. 

There could also be existing professional networks that you are a part of that allow you to 
network and form collaborations with community-based organizations (CBOs). You 
could be part of the associations or groups based on your interests, such as a volunteer in 
a non-profit organization, at a food bank, or as a member of an interest group such as bird 
conservation society, a band, a choir, a sports team, a book club, and so on. These personal 
and professional networks might give you opportunities to connect with other collaborators, 
researchers, and community members passionate about your research projects and interests. 
It is also possible that the communities themselves approach a researcher to be part of their 
research team or that the researcher is invited to join an existing interdisciplinary team or 
initiative. 

Working with interdisciplinary teams means taking an inclusive approach to defining the 
key social issues to be addressed. In identifying the topic, researchers should incorporate 
community perspectives, relevance, and understanding of the issue at hand beyond academic 
definitions, terms, and concepts. It is important to understand the problem that is affecting 
the community from the perspective of community members themselves. The stakeholders 
collaborate to identify the problems and fine-tune the goals by making visible the historical 
contexts that shaped the social inequities. For instance, if the intervention is about effective 
prevention of flu infection through vaccination, then researchers should spend some time as 
they design their intervention understanding how the community has historically dealt with 
viral infections, what were some of the previous programs and interventions conducted in 
the community, and what were previous experiences of the community with social science 
researchers. 


Establishing Partnerships with Relevant Stakeholders 


Once the topic is determined, researchers need to identify the relevant stakeholders and 
form partnerships with community-based organizations who are also committed to the same 
social justice goals as the researchers. It is important to form coalitions and partnerships with 
community-based organizations who share your vision for the program and are also commit- 
ted to the same social justice goals that you are. Working with diverse organizations allows 
researchers to consider diverse perspectives that might not be immediately obvious to them. 
There could be several barriers for the intervention that researchers might not have consid- 
ered. It also provides an opportunity for community members to express their voice and feel 
heard. These community organizations can also help to mobilize members to participate in 
the research process. The downside of involving stakeholders is that the planning process will 
take longer to complete. So, the researcher needs to make some trade-offs depending on the 
specific context and social issue that they are studying. 

Community-based organizations and stakeholders can include a wide range of people and 
organizations such as potential participants, family members, teachers, employers, cowork- 
ers, community leaders, government representatives, authority figures, media outlets, social 
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advocates, and funding agencies. Community-based organizations often work with a range of 
service providers and experts such as legal experts, counselors, health-care providers, research- 
ers, and educators, depending on the specific issue that they address. For example, in a school- 
based bias reduction intervention that the second author is involved in, the research team 
consists of faculty and staff (from various disciplines such as Communication, Psychology, 
Performance Studies, and Higher Education), instructional technology experts, pre-service 
teachers, Center for Teaching Excellence staff, school administrators, middle school teachers, 
parents, school students, and counselors. Projects relating to health communication interven- 
tions could involve public health researchers, health communication scholars, community 
health workers, social workers, hospital patients, clinic staff, nurse practitioners, physicians, 
dieticians, and so on. It is important, then, for researchers to avoid approaching a partnership 
by thinking that they, themselves, possess all the knowledge or that they are doing the com- 
munity a favor by working with them. 

Setting up a community advisory board (CAB) is a common research practice in 
projects that involve participatory methods. Such boards should be diverse and include rep- 
resentatives from various backgrounds and strata within the community. Developing strong 
and trusting relationships from the beginning of the project by involving the main stakehold- 
ers in the process is especially helpful in dealing with any difficult challenges in the research 
project. Taking a collaborative participatory approach to community-based research could 
mean involving community members of the advisory board to help with any number of top- 
ics including: 


e Identifying the goals, defining the problem, and fine-tuning the research questions 

e Selecting strategies to achieve the identified priorities and goals associated with the 
intervention 

e Recruiting research participants who might otherwise be hard to reach 

e Formalizing roles and responsibilities for various team members involved in the research 

e Setting priorities for what the research can address based on needs assessment and shared 
resources available 

e Identifying potential barriers to implementing the research project 


It is crucial to consider what skill sets each team member brings to the table, what is at 
stake for them in the research, and how they might be approaching the problem at hand. 
In other words, community-based research values multiplicity or triangulation of per- 
spectives, interests, and skills. Several partnerships evolve with time by building trust 
between researchers and community members. As Billard (2019) points out within the 
context of public scholarship with the National Center for Transgender Equality, doing 
participatory action research often means that researchers need to rethink the meaning 
of research, what skills they bring to the table, what counts as research, and how success 
is measured in that context. The participatory nature of such research means that just 
as academics offer new perspectives and knowledge to communities and publics, they 
should also be open and willing to learn about perspectives from non-academic com- 
munity members. 

If they are new to a community, researchers often rely on informants to gain better 
understanding of the cultures they are examining. Informants are members of the culture 
or community who might serve as bridges between the researcher and the community 
because of unique insights into the culture that, due to their positionality, they can provide. 
There could also be gatekeepers such as community leaders who might grant access to 
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the research space and setting. It helps to work through existing community-based partners 
and networks to bring about maximum impact on the social causes that the researchers are 
passionate about. 

Deliberative democratic processes in design and implementation of research serve as mech- 
anisms for promoting shared ownership, inclusive decision-making, and reconciliation of dif- 
ferences. They focus on dialogical models to discuss and resolve differences in perspectives, 
values, and desires of various stakeholders. Whenever possible, it would help researchers to 
find ways to encourage co-creation of research procedures with community members. For 
instance, Benjamin Bates and colleagues (2019) discuss the idea that health interventions often 
assume community members as passive recipients of benevolent charity of the researchers 
rather than seeing community members as active participants and knowledge-producers in 
the research process. 


Research in Action 


A strategic tool that can help in community-based projects is a community asset 
mapping exercise. In this section of this chapter, we will guide you through the 
steps to work on a community asset mapping exercise that you can conduct for any 
community or group with which you will be collaborating. The community map- 
ping tool is part of a methodological and theoretical approach called Asset-Based 
Community Development (ABCD) model for sustainable community development. 
This is a grassroots and participatory approach. It focuses on using existing com- 
munity assets to bring about transformative change. It was developed by Kretzmann 
and McKnight (1993). The Asset-Based Community Development (ABCD) 
model has six main assumptions: 


e Everyone has gifts and assets to contribute. 

¢ The focus should be on community strengths rather than needs. 
e Existing resources can be mobilized toward community-building. 
e Relationships build strong communities. 

e Relationship-building occurs through active listening. 

e Change should be community-driven and citizen-centered. 


In the mapping exercises, one can map any number of individual and community-level 
assets as well as their connections and relationships across a geographic space. The map 
includes any number of assets such as talents, skill sets, networks, knowledge, money, 
time, space, and stories. 

Figure 11.4 shows one possible sample map that could be helpful for you to under- 
stand how a community asset map could look like for a specific community and issue 
of interest. 
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Hobby Clubs 


Figure 11.4 Community asset map 


Source: Emily Riewestahl 


Asset mapping can be done in multiple ways. Here is one way to approach asset mapping: 


e Establish the geographic boundary for the map and the region that you will focus on. 

e List all the assets (skills, talents, capacities, passions, and resources) in the com- 
munity that are relevant contributions to the research topic. For example, your list 
could include physical spaces (such as parks and transportation systems), economic 
assets (such as banks, grocery stores, and local businesses) and social networks (such 
as clubs, associations, cultural groups, and communication channels). 

e Gather stories and narratives relating to the assets through archival research, oral 
histories, in-depth interviews, and focus group discussions to understand the con- 
text, background, and history of the research topic of interest. 

e Note where there are gaps and strengths. Visually organize this information in an 
accessible and meaningful way to present it back to the community as a means of 
leveraging assets to bring about change. 
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Developing Research Materials for Community-Based Projects 


Pretesting of communication materials is the next step, often referred to as formative 
research. In this stage, the research materials are tested to make sure that they are appropri- 
ate. The implementation of the intervention involves decisions such as what should be done, 
when, how, where, and using what resources. For example, if designing an online media 
literacy intervention for middle school students on prejudice reduction, the pretesting stage 
might involve coming up with scripts for possible scenarios that describe discrimination in 
school-based settings. Since it would be expensive to create video-based online media literacy 
modules, which would involve scriptwriting, filming, and editing, it is best to conduct some 
formative research to make sure that the scenarios being developed are culturally relevant, 
age appropriate, and credible. Such inputs from community members at these initial stages 
of the research process could not just help the research team save on resources but also serve 
as an opportunity to involve community members in meaningful ways as part of the research 
process in co-creating research materials. Once the researchers receive feedback from the 
pretesting stage on initial research materials, researchers often conduct a smaller pilot study 
at this stage by implementing the intervention on a small subpopulation before launching it 
fully on a large scale. At this stage, the input from the pilot study can still be used to make any 
needed corrections and changes. 

Designing successful community-based communication interventions means listening to 
the community and their needs at every stage of the research process. Dutta (2008) refers to 
this as taking a culture-centered approach to participatory research with communities that 
centers on cultural humility and listening. The researcher needs to be sensitive to the unequal 
hierarchies and status differences between the researcher and the researched. It is important 
for researchers to be familiar with the subculture, language/dialects, and communication 
preferences of participants. In working with participants in the Mexico-U.S. borderlands in 
the South Texas colonias, for instance, it could mean conducting research in Spanish using 
face-to-face interviews rather than using phone calls or online surveys. Using inclusive and 
nonjudgmental language to refer to participants and their situations goes a long way in mak- 
ing them feel included and acknowledged. It means being explicit about recognizing cultural 
differences and social hierarchies to ensure that power differentials are acknowledged and 
discussed. Researchers should avoid using a deficit model that assumes that marginalized 
groups are to be blamed for their conditions and also be careful to avoid biologically essential- 
ist theories that suggest that socio-cultural inequalities exist due to personal deficiencies or 
cultural inadequacies. 

Researchers should prioritize jargon-free language in their materials. In designing ques- 
tionnaires and surveys, for instance, the language of research, including consent processes, 
can often be highly technical, hard to read, and difficult to understand. For example, Harlyn 
Skinner and colleagues (2015) discuss how their research team originally used technical jar- 
gon in their study without recognizing the importance of more accessible language. With the 
purpose of strengthening trust between the research team and members of the community, 
they changed their study materials to say that they want to “learn more about genetic factors 
relating to health disease” instead of the previous consent form, which used words such as 
genomic signatures, responsiveness, interventions, and CVD disparities. 

Beyond language, it is also important to consider the mode or format used to deliver 
questionnaires and other measurement instruments. Universal design principles should be 
considered in terms of making sure that everyone who would like to access research materi- 
als can do so. For example, using online surveys as well as printed ones in large font allows 
participants the flexibility to determine which mode is convenient for them to share their 
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information. Even small changes such as using simplicity in design, pre-testing materials, 
giving detailed instructions, offering the materials in multiple languages and formats, and 
minimizing fatigue are important considerations. 


Implementation of Community-Based Research Projects 


In this section of the chapter, we will focus our attention on some of the practical steps 
involved in implementing community-based research projects. We cannot emphasize enough 
the importance of having clear research protocols and shared agreements to be devel- 
oped collaboratively as a team. This means that the nature of the relationship between the 
researcher and the participants, responsibilities, time commitment, and division of labor 
should be clarified ahead of time and throughout the research process. It would be important 
to consider how to make decisions, how to interact and communicate with one another, and 
what values are shared. Interpersonal and inter-organizational issues such as disagreements, 
refusals, noncooperation, and dissent are common in community-centered research and need 
to be addressed with transparency and fairness. For instance, community members and indus- 
try collaborators might have different scheduling expectations and work ethics compared to 
researchers from academe. Therefore, it is vital to have discussions on these issues as part of 
the implementation of the research project. 

It is important for researchers to discuss with community members practical barriers 
such as accessibility, transportation, time availability, and financial costs that come in the way 
of research participation. Some challenges could relate to difficulties in getting to the venue; 
arranging for alternative care for loved ones such as elders, pets, or young children; and find- 
ing time between multiple jobs and roles to participate in the research. For instance, for the 
second author’s Media Rise project, she often needs to locate free, accessible, convenient, and 
safe spaces for community members to meet. She also plans for transportation to and from 
the venue, childcare support for parents, refreshments for attendees, and access to technology. 

Given the inherently participatory nature of community-based research studies, research- 
ers who use methods such as surveys or field experiments, for instance, should consider 
using community-based organizations and other networks to reach out to groups that are 
traditionally left out of data collection. As we have seen in Chapter 4 on sampling, inclusive 
approaches to sampling are not just important in community building and action-orien- 
tation in social justice projects, but they also help increase generalizability, representativeness, 
and external validity of the research study. It is important to understand cultural expectations, 
social norms, and power differences in terms of who among the community has voice, power, 
and access to participating in research in active ways. Factors such as severe weather, religious 
holidays, migratory patterns, and political climate can significantly influence sampling. Quan- 
titative critical researchers should find ways to overcome these barriers to allow for a greater 
number of community members to share their perspectives in the researchers’ studies. 

Since community-based research is sometimes long-term in nature, community consent 
is negotiated multiple times from community members before, during, and after the research 
project. Informed consent is vital to obtain from community participants before the start of 
the research project, as it helps to communicate and set reasonable expectations. Community 
members should be involved in determining risks, harms, benefits, discomforts, and adverse 
effects. Working in collectivist communities could mean that familial, tribal, and other social 
affiliations take precedence over individual preferences in influencing decisions about research 
participation and the consent process. Informed consent, under such circumstances, might 
not necessarily be discussed only at the individual level but also with the community as a 
whole at multiple stages of the research. For example, while conducting research with first 
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peoples and tribal communities, it is important to preserve and honor indigenous protocols, 
including getting community consent from intertribal IRB ethics boards (Kelley, Belcourt- 
Dittloff, Belcourt, & Belcourt, 2013; Lambert, 2014). 

When working with stigmatized communities who are often invisible and underrepre- 
sented in academe, it is important to make space for the voice of community members 
to be heard within scholarship. Identifying and bringing attention to community members 
through research could serve as a powerful way of acknowledging and validating the wisdom 
and expertise within historically marginalized communities. However, naming community 
experts publicly in the research could also open up the possibility of gaining negative atten- 
tion, which could lead to challenges such as conflict, embarrassment, threat, withdrawal of 
resources, and injury. Moreover, researchers have to be careful not to speak for others or 
inadvertently represent minoritized voices from a position of superiority. Therefore, decisions 
about whether participants should remain anonymous, be addressed using pseudonyms, or be 
explicitly identified have to be made with great care and thought, balancing risks of potential 
harm with benefits that come from recognition, validation, and affirmation of community 
expertise. 


Evaluation and Feedback in Community-Based Research Projects 


In applied communication contexts, social justice researchers are often interested in assess- 
ing the impact of interventions in the real world. We could think of any community-based 
research program or intervention as a field or natural experiment (see Chapter 6 for experi- 
ments). Using a pretest/posttest experimental design model is quite popular. This often means 
using time-series longitudinal assessment to monitor the effects of the intervention on key 
outcome variables. The independent variable is the program or intervention or treatment. An 
intervention could be a media literacy program conducted in middle school, a health cam- 
paign on HIV/AIDS prevention, or a diversity training program for employees. The depend- 
ent variables are the effects and outcomes on participants. These could be a range of variables 
such as knowledge, attitudes, beliefs, and behaviors that are affected by the intervention (for 
example, changes in critical media viewing after a media literacy intervention, adoption of 
safe sex practices after being part of a HIV/AIDS prevention campaign, or more inclusive atti- 
tudes after being part of a series of diversity training workshops in an organization). However, 
it is not always possible to conduct a pretest/posttest design field experiment nor always fea- 
sible or practical to have a control group. In these cases, even a single group post-only design 
could help evaluate the effectiveness of a community program in reaching its goals, but it is 
not possible to establish causality using such quasi experiments and pre-experimental designs 
(see Chapter 6 on experiments). 

Researchers should have an evaluation plan for assessing whether the community-based 
project has reached its intended goals and aims. Taking a participatory collaborative approach 
to evaluating community-based projects means that the evaluation plan reflects the goals and 
needs of community stakeholders. The evaluation team can involve a large team including 
such people as researchers, teachers, parents, administrators, social workers, policymakers, 
government officials, technicians, and professionals, depending on the nature of the project. 
A common misconception among many is that evaluation is conducted only after a program 
has been implemented to check whether the outcomes have been achieved. Yet, as we have 
established, evaluation is often done at the initial formative research stage, as well. 

An important aspect of evaluation of community-based research projects is bringing the 
data back to the community and members of the partnership to get their inputs and collec- 
tively discuss and interpret the findings. This step is referred to as participant or member 
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checks, when researchers include community members and research participants within the 
research process by giving them drafts of their research reports, notes, and transcripts to 
receive feedback from them. This step is especially helpful with social justice research, which 
often deals with sensitive information, to make sure that there are no misrepresentations or 
misunderstandings relating to the data that are being collected and distributed. 

The evaluation plan should include program outcomes that are identified as goals to be 
achieved. It is important to think about the outcomes at the individual as well as commu- 
nity level. These could include short-term or intermediate-term goals such as increased self- 
efficacy, individual-level outcome measures, and shared research partnership. They can also 
lead to long-term changes such as policy changes, shared power, capacity building, greater 
equity, more access to resources, and other community-level measures of well-being and 
social transformation. For each of these goals, it would help to articulate how and why they 
are important and what the desired outcome would look like. 

Researchers could use information from many different sources to collect data that could be 
relevant in designing, implementing, and assessing community-based projects. Some sources 
of information could include participants in a community-based program, staff and leaders 
within community-based organizations, researchers and experts on the topic, and other sig- 
nificant community stakeholders. Depending on the project and resources available, here are 
some types of assessment information that could be collected from various sources to evaluate 
the effectiveness of the community project: 


e Artifacts and archival data (such as newspapers, yearbooks, syllabi, etc.) 

e Internal reports of community-based organizations 

e Publicly available records 

e Community-level data (may or may not be publicly available) 

e Similar programs or initiatives conducted in other communities 

e Published research on the topic, including measures and tests used in prior studies 
e Social media networks such as Facebook and Twitter 


In Chapter 10, we discussed the importance of multiplicity of methods and the need for tri- 
angulation. Researchers should consider multiple measures and methods for the same source 
of information. Depending on the research questions and stage of the project, they could use 
focus group discussions, surveys, or in-depth interviews with various groups such as potential 
participants, program participants, staff members, and other stakeholders. For instance, they 
could do an audit of internal records in conjunction with in-depth interviews with program 
staff. Community organizations often do not have large amounts of resources for research. 
Therefore, researchers should strive to use only those measures and methods that would be 
helpful, valid, reliable, and credible. When there are multiple team members and stakehold- 
ers, there could be a tendency to include measures that are only tangentially helpful to the 
research questions and goals. In terms of quantitative methods, most interventions set out 
with clear and measurable objectives, and therefore, the outcome variables are compared to 
baseline pretest measures to keep track of how the intervention is working for the community. 


Making Research Findings Available to Communities 


Communities should also be incorporated into the distribution and dissemination of research 
results. Ideally, findings should be made publicly accessible through multiple formats such as 
photo exhibitions, pamphlets, blogs, websites, panel discussions, newspapers, radio channels, 
social media, etc. to highlight social issues to community members, policymakers, change 
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Research in Action 


Here are some helpful web resources relating to program evaluation and assessment of 
any community-based initiative. Although many of them are from health communica- 
tion contexts, they can be applied to any topic or community context: 


1. American Evaluation Association (AEA) (www.eval.org/): This organization 
serves as a hub for evaluation research in a variety of contexts such as collaborative 
and participatory evaluation, health evaluation, democracy and governance, disa- 
bilities and underrepresented populations, higher education, crime and justice, data 
visualization, environmental program evaluation, feminist issues, and indigenous 
people in evaluation. It has an online public library and also offers a tip a day for 
members. 

2. The Centers for Disease Control’s Performance and Evaluation Office 
(www.cdc.gov/eval/framework/index.htm): This office provides important toolkits 
and resources for developing effective program evaluation in public health related 
contexts in a systematic way. It includes the “Framework for Program Evaluation in 
Public Health” and standards for effective evaluation of health interventions. 

3. School-Based Interventions (www.interventioncentral.org/): This site provides 
guidance, toolkits, measures, and manuals for researchers, teachers, and school 
administrators designing interventions relating to literacy, childhood health, bully- 
ing, and other topics related to children and youth. 

4. Office of Adolescent Health (www.hhs.gov/ash/oah/): The U.S. Department 
of Health and Human Services has an office devoted to adolescent health that 
offers several examples of programmatic initiatives, evaluation, research, resources, 
and training relating to such topics as teen pregnancy, teens and tobacco use, teen 
mental health, and teens and sexual health. 

5. Domestic Violence and CPBR Resources (https://cbprtoolkit.org/): For 
those committed to addressing domestic violence related issues in the community, 
this web resource has tools and resources. These tools are tailored toward emerging 
scholars working in community, practice-based, academic, and policy settings. 

6. The Association of Asian Pacific Community Health Organizations (AAP- 
CHO) (www.aapcho.org/resources_db/cbpr-toolkit/): In collaboration with the 
National Association of Community Health Centers, this organization provides 
resources to researchers interested in working with medically underserved popula- 
tions, such as Asian Americans, Native Hawaiians, and other Pacific Islanders. 


agents, government officials, and other stakeholders who could help to reduce inequalities. 
From a social justice perspective, community-based research projects are action-centered. 
Therefore, research does not end simply with journal articles, conference papers, or reports 
that summarize findings. The research needs to bring about transformative action and help 
address the community-identified problem. In other words, the research often serves as a cata- 
lyst to bring about new actions, plans, policies, and strategies for social change. Communica- 
tion scholars might also consider going beyond traditional academic conferences and journals 
to bring research to relevant stakeholders outside the discipline and academe. 
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An important consideration is whether the community members are to be considered 
authors or participants, depending on their level of involvement in the research process. 
Good research authorship practices require that authors be involved in design, data collection, 
writing, revising, and final approval. Those scholars who value relational ethics might reject 
hierarchical notions of what it means to be an expert. They could do so by involving research 
participants at several crucial decision-making stages, including and even as co-authors. This 
could mean publishing their work in open-access journals or making their research available 
through op-ed pieces or blog posts that are free to access. In the next chapter of this book, 
you will learn more about the role of quantitative research in policy-making and the ways in 
which one can translate research for a more general audience to make research-based insights 
available in multiple, alternative, and accessible formats to the public. Chapter 12 will go 
more in depth about various ways in which researchers could make their research available to 
a broader public. 


Practical Considerations in Community-Based Research Approaches 


Community-oriented participatory research involves long-term time investment and some- 
times also financial commitment, which could also affect work-life balance, health, and per- 
sonal relationships. Quantitative criticalist scholars might run into dangerous and challenging 
situations as part of their activism that could involve direct violence or threats against them. 
Resisting the status quo means challenging authority figures and those in positions of power. 
For instance, if you are a researcher studying LGBTQ+ rights or reproductive justice, it is not 
uncommon to run into challenges both from the community that you are studying and even 
perhaps the institution that you are affiliated with. Community-based social justice researchers 
could experience serious physical, emotional, and psychological discomfort and stressors. It 
could mean putting physical bodies on the line at the forefront of protests. 

Despite researchers’ best efforts, it is not possible to anticipate all the risks and benefits 
to the participants. Therefore, it is equally important to be ready for unanticipated research 
experiences that take researchers by surprise and need to be addressed through flexibility 
and reflexivity. Emergent research design is especially helpful while conducting social justice 
research in partnership with communities, where there could be unanticipated (or even antic- 
ipated) changes to the research design that are responsive to the changing needs, expectations, 
and goals of the community or organization with which the researchers are collaborating. For 
instance, the staff turnover within social justice community organizations could mean having 
to start afresh with establishing relationships with the organizations. Similarly, if community- 
based participatory research involves service learning using student researchers, there could be 
turnover in terms of new students each semester who need to be trained again to work with 
a specific community-based project. 

Nevertheless, with the right type of support, quantitative criticalist scholars can bring about 
meaningful and significant transformative change through community-based applied Com- 
munication research. Social justice scholars could champion the use of research findings to 
uplift and serve the communities with which they work. They can help bring about policy 
changes, build capacity, and harness existing resources within the community to address long- 
standing challenges. They could also assume an advocacy role to encourage the utilization of 
research results to develop long-term policies and programs for the betterment of the com- 
munity and human condition at large. Indeed, community-based social justice research often 
blurs the boundaries of research, teaching, service, and community work. When done right, 
it can serve as one of finest examples of transformative and impactful scholarship, which is not 
just necessary but vital for society. 
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Scholar Spotlight: Dr. A. Susana Ramirez 


Figure 11.5 Scholar Spotlight: Dr. A. Susana Ramirez 


A. Susana Ramirez is Associate Professor of Public Health Communication at the 
University of California in Merced. In 2019-2020, she was Visiting Professor at the 
Nacional Autónoma de México (UNAM), through the University of California Edu- 
cation Abroad Programs. She is a social scientist whose research at the intersection of 
communication science, public health, and Latinx health focuses on communication in 
the service of social justice and health equity. 


Q: Could you please share a little bit about your journey into community-based research? Was 
there a specific moment when you decided that this is what you wanted to do? 

Dr. Ramirez: My commitment to community-engaged research stems from my dis- 
satisfaction with large national datasets to answer my research questions. In my 
doctoral training, I learned about race in the media and health disparities and the 
ways in which communication could both contribute to disparities but also help 
these, and I started asking questions based on my own experiences as a Mexican 
child raised in the United States. What I found was that most of the research 
either ignored ethnic identity or focused on the challenge of linguistic translation 
for Spanish-speaking populations. Based on my own experiences, I knew this was 
incomplete. But when I used a variety of national datasets to test hypotheses about 
the nature of the acculturation process—and how that affects the way Latinx 
perceive health information as relevant—I realized how limited the “nationally 
representative” data were. The results didn’t make sense: They were inconsistent 
and counterintuitive. So coming out of my postdoctoral training at the National 
Cancer Institute, I looked for a faculty position where I could connect with a 
diverse Mexican American population to help me refine and test my hypotheses. 
Today I sit in a department of public health sciences, and I am interested more 
broadly in the transnational flows of communication, people, and products and 
population health and doing community-engaged research with a population that 
lives this transnational life helps to make my work more relevant than ever. 


Community Partnerships 319 


Q: There are many related terms such as community-based participatory action research, 
intervention research, participatory action research, social justice scholarship, activist schol- 
arship, engaged scholarship, translational scholarship, and so on within Communication. 
Which of these terms do you most closely identify with and why? 

Dr. Ramirez: I describe myself as a scholar committed to social justice. I practice 
community-engaged scholarship, which I distinguish from community-based 
participatory research (CBPR). CBPR really has to be driven by the community, 
whereas community-engaged scholarship is in partnership with and sensitive to 
the community, but it can be investigator-driven. My goal is to advance the sci- 
ence of communication—so that practitioners of public health communication 
can “do” communication better and so that the development of policies and sys- 
tems that create a just world will be informed by science. 

Q: What are some of the main challenges in doing community-based research? What can be 
done at the disciplinary and institutional levels to better support this type of scholarship? 

Dr. Ramirez: Community-engaged research is time-consuming and political in a 
way that traditional scholarship is not, and both of these differences can have 
material consequences on junior scholars’ career advancement. For example, to 
do good community-engaged research, you need to be authentically engaged in 
the community. That means you show up to community events, you participate 
on local advisory boards and councils, and you help community organizations 
even when you aren't directly benefiting from the activity. All of these activities 
are time-consuming and represent time that a researcher could have spent writ- 
ing a paper that will be helpful for promotion. But also, the authentic relation- 
ships take time to develop. You can’t expect that most communities—especially 
if you work with marginalized groups—will immediately grant you the kind of 
entry that you need to do meaningful work. You need to earn their trust over 
time. Again, that time—and it might be years!—means that you aren’t doing the 
research and publishing that you need to be doing to advance on the traditional 
academic timeline.In terms of the politics of community-engaged research, the 
world gets very small when you start interacting with a community. There are 
informal and formal leaders with long histories in any community, and you 
want to be very careful as you start getting to know folks so that you don’t end 
up unintentionally aligned with factions that might make it difficult for you to 
proceed in the future. 

Q: What practical advice and tips do you have for budding scholars interested in community- 
based participatory research? 

Dr. Ramirez: Don’t do it till you have tenure! I’m only kind of kidding—scholars 
who are interested in community-engaged work face real barriers to promo- 
tion and tenure in the current system. But if junior scholars are committed to 
this work, then there are some tips I can offer. First, don’t try to do everything 
yourself. Partner with someone at your institution who has longstanding relation- 
ships with the communities you want to work with. If they have existing data, 
see if you can use it in a new way, but make sure it is relevant for your own body 
of work. That both helps with building the relationships by showing your com- 
mitment to the community and keeps you looking productive on the CV. If that 
doesn’t exist, then you’re going to have a harder time. Find existing data sets that 
you can keep using to publish—and share these studies with the community you 
are getting to know and be known by. 
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Community-based projects are common in social justice research, although they are still 
somewhat rare within Communication as a discipline. Community-based research projects 
vary greatly in terms of topics, communities, level of community involvement, and types 
of methods used. They often use mixed methods approaches that combine quantitative and 
qualitative methods such as participant surveys, in-depth interviews, focus group discussions, 
archival methods, and field experiments. Community is typically understood to be participant 
groups outside of academe. They can include both “real world” and virtual communities such 
as residents of a local area, government agencies, businesses, professional networks, schools, 
libraries, interest groups, volunteer organizations, and so on. 

Community-based projects tend to be problem-centered, participatory in nature, and 
action-oriented toward transformative social change. Taking a socio-ecological approach, 
social justice researchers often need to influence several layers or spheres of influence from 
micro-level individual factors to macro-level societal factors. There are many different types 
of community-based research approaches including community-based participatory research, 
participatory action research, trauma-informed participatory research, program evalua- 
tion research, engaged scholarship, and public scholarship. Community-based participatory 
research (CBPR) involves the community throughout the research process on socially-rele- 
vant issues through participatory methods and shared decision-making. Participatory action 
research (PAR) is similar to CBPR but is more explicitly action-oriented and critical in its 
approach in terms of bringing about social transformation. Program evaluation research stems 
from field experiments and intervention sciences, focusing on organized programmatic initia- 
tives with shared goals. Engaged scholarship refers to partnerships and engagement formed 
and maintained between researchers and non-academic stakeholders in the research process. 
It need not be action-oriented or social justice-oriented, unlike PAR, which tends to be 
explicitly social justice-oriented. Public scholarship focuses on public input and outreach of 
research to wide groups of citizens. 

The level of participation of communities could vary within research collaborations. Com- 
munity-oriented research projects are oriented toward addressing community issues but typi- 
cally do not involve community members. Community-placed research projects are conducted 
within community settings, and the community is minimally involved in research processes. 
Their largest role is as participants. Community-based participatory research is typically set 
within communities and also involves community in various stages of the research project. 
Finally, community-driven research projects are those where community members take the 
lead in the research process. 

Typically, in community-based research projects, after determining the initial topic of inter- 
est, the key stakeholders are identified and community partnerships are formed between 
researchers and community-based organizations. It is also possible that community-based 
organizations already have some programmatic initiatives in place that are evaluated and 
assessed by the researchers. Problem identification and understanding of key concepts is done 
collaboratively through locating and summarizing relevant bodies of both formal and informal 
knowledge. 

Research design, study materials, and evaluation are conducted with cultural sensitivity 
in collaborative ways, with the aim of taking an action-oriented programmatic approach to 
social justice issues. Inclusive methods of recruitment and retention of study participants, 
building trust and rapport, making long-term commitment, and emphasizing multiplicity of 
methods and viewpoints are emphasized in this approach. Pretesting and pilot-testing of pro- 
gram and research materials are aspects of formative research to get input early in the research 
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process. Evaluation of community-based programs needs to clearly articulate program out- 
comes in the short and long term at the individual and community level. Several sources of 
information such as public records, newspaper articles, social media, interview data, surveys, 
feedback forms, and the like can help assess whether an initiative or program is creating a posi- 
tive change, as intended. Project materials and goals can be fine-tuned and improved based on 
evaluation research. Research findings are ideally shared in multiple, alternative, and publicly 
accessible formats such as opinion pieces, blogs, social media posts, newsletters, photography, 
performances, brochures, documentary films, and the like, beyond traditional publications in 
scholarly outlets such as journals and conferences. 

Although community-based research involves many challenges and potential risks such 
as long-term involvement, emotional stress, physical discomfort, and unanticipated research 
experiences in the field, these things can also be extremely meaningful, impactful, and trans- 
formative, bringing about long-term positive change. 


Discussion Questions and Activities 


1. Lets focus on a social justice topic that interests you. In a sentence, explain the main 
topic that you would like your research to address. Now determine who would be the 
key stakeholders with whom you would have to collaborate and form partnerships to 
effectively address the topic that interests you. Next, what are the resources available to 
you that you could leverage? These resources could be personal knowledge and skill sets, 
or more broadly they could be other community resources, funding opportunities, and 
professional/personal relationships to which you have access. 

2. How isan action-oriented approach different from other approaches to research? In what 
ways can the community be involved in understanding and defining the key issues at 
stake? 

3. Find an existing traditional research study done in a university setting such as a lab- 
based experiment, content analysis, or survey methods. After reading this chapter, how 
would you redesign the same study if you were to address this same research question in 
a broader community context with non-academic stakeholders in mind? What would be 
some important changes you would make to the design? 

4. Keeping the principles of mixed methods research discussed in the previous chapter in 
mind, pick two or three methods (such as surveys, experiments, content analysis, arts- 
based research, archival research, textual analysis, focus group discussions, ethnography, 
and so on) that would be most helpful to you for a community-based project that you 
are interested in. Why did you choose these methods, and how will they work together 
within the specific community and topic that interests you? 

5. Imagine that you have been invited by a local community-based organization to serve as 
a research consultant. They are going to launch a sexual violence prevention campaign. 
Using the information, you have learned in this chapter and elsewhere in this book, 
design a pretest/posttest field experiment to evaluate the success of the program. What 
are some considerations that come to mind as you design such a study? 
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12 Communicating Research for 
Publication, Policy, and the Public 


In this book, we have discussed how to design and carry out a quantitative research study with 
a critical consciousness, taking care to be inclusive, ethical, and just. Equally important for the 
potential of quantitative social science to harness the “power of numbers”—to identify and 
help address social injustices and imbalances in power—is how that research can be used. The 
researcher concerned with social justice ideally not only designs and carries out research with 
a critical consciousness but also facilitates the application of that research in direct efforts to 
intervene in social issues and inequalities. 

Engaging in research for social justice does not mean taking shortcuts in the standards and 
practices associated with quantitative methodology. To the contrary, in order for quantitative 
research to be of use in social justice pursuits, it must be sound social science. To that end, the 
research will typically follow the procedures associated with publication. Published research is 
understood to have withstood the rigorous process of scientific peer review. Therefore, pub- 
lished studies are generally considered more authoritative compared to unpublished studies. 

Yet, if researchers only pursue the publication of their research in academic journals and 
other scholarly outlets, the utility of that research for real-world problems and possibilities is 
likely to be quite limited. For the most part, the readers of academic journals and other schol- 
arly outlets are academics themselves, rather than policy makers, members of nonprofit or 
community-based organizations, and other potential change makers. Therefore, researchers 
committed to social justice are often motivated to do what they can to get research into the 
view of members of the public and/or into the hands of change makers. 

In Chapter 11, we explored community-engaged research, one avenue toward producing 
research that has a direct bearing on real-world issues and concerns. In this chapter, we dis- 
cuss two other avenues: writing for the general public and writing for the purpose of shaping 
policy. Given the importance of publishing research, too, however, for both the status of many 
researchers themselves and the stamp of approval such a process signifies for the research, we 
begin with a detailed discussion of writing for scholarly publication. 

Across all three of the types of writing discussed in this chapter—for publication, for the 
public, and for policy—one fundamental goal is the same: clarity. In this chapter, we empha- 
size clear and precise writing for the three purposes we have outlined. 


Writing for Publication 


Publication is typically the primary goal of scholarly research for a number of reasons. First, 
publishing research is an important way in which scholars exchange ideas and findings with 
one another. Through this process, knowledge is created and disseminated, building an ever- 
increasing body of evidence. In other words, we develop what we know about topics and 
themes in large part because of the availability of the scholarly evidence associated with 
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those topics and themes. We cannot learn from research that takes shape only as a file on the 
researcher’s computer or an idea in a researcher’s mind. Rather, that research has to see the 
light of day in some type of published form. 

Second, in many cases, researchers who are associated with colleges and universities some- 
times must publish in order to obtain job security and advance their careers within the exist- 
ing reward structures. At many colleges and universities, research must be published in order 
for the researcher to secure tenure and promotion, since publishing research demonstrates that 
the researcher is making an active contribution toward the production of knowledge. Tenure, 
in turn, protects the researcher, for the most part, by providing a secure position in which the 
researcher can enjoy academic freedom. 

Academic freedom means that one is able to pursue research and teaching without fear 
of punishment, censorship, rebuke, or retaliation. A fundamental principle in the creation 
and distribution of knowledge, academic freedom ensures that academics can pursue even 
ideas that are controversial, are not in the interest of colleges or universities themselves, or 
otherwise push against dominant systems and structures. This principle is especially important 
for social justice research since such research is positioned in opposition to power structures 
that promote and protect injustices and inequalities. Ideally, a university-based researcher 
would immediately and automatically enjoy academic freedom, but sometimes the researcher 
does not perceive or experience such freedom before tenure or some other long-term com- 
mitment to employment is established. The American Association of University Professors 
defines academic freedom as “the indispensable requisite for unfettered teaching and research 
in institutions of higher education” (www.aaup.org/our-work/protecting-academic-free- 
dom). Its policy document states: 


Institutions of higher education are conducted for the common good and not to further 
the interest of either the individual teacher or the institution as a whole. The common 
good depends upon the free search for truth and its free exposition. 


(1940 Statement of Principles on Academic Freedom and Tenure, www.aaup.org/our-work/ 
protecting-academic-freedom) 


The third reason to seek publication of research is that the publication process ideally func- 
tions as a way to separate out more credible evidence compared to less. As you will read, 
published work has undergone close and careful scrutiny before making its way to the page. 
This scrutiny both recognizes strong research and helps create it, since the feedback obtained 
when a paper is under review for publication can provide guidance for improvement. 


Forms of Academic Writing 


Once you have designed and carried out a study, there are a number of forms the resulting 
written report of that study can take, even within the category of writing for publication as 
opposed to writing for policy or for the general public. Common forms include conference 
papers, posters, and/or presentations; book chapters; books; and articles in academic journals. 
We will briefly discuss each, although our emphasis will be on journal articles, since that tends 
to be the most frequently chosen avenue for publication among quantitative social scientists, 
including those in Communication. 

Many times the first step to take upon the completion of a research study is to submit 
that study to be considered for selection for presentation at a conference. The Interna- 
tional Communication Association, the National Communication Association, and the other 
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disciplinary and co- or cross-disciplinary organizations that we have discussed throughout this 
book typically convene an annual meeting, at which new research is presented to an audience 
of conference goers largely consisting of colleagues in the field. This is a wonderful oppor- 
tunity to network and exchange ideas, research approaches, and findings among those with 
shared interests, and the feedback on a study received through this process is often used as a 
step in strengthening that study prior to the pursuit of publication. 

Indeed, a research paper submitted to a conference is often quite similar in nature to a paper 
submitted for consideration for publication in a journal. It will traditionally have the same 
elements: abstract, introduction, literature review, methods, results, and discussion, as well as 
references and, frequently, tables and figures (all of which are outlined in detail later). Often, 
there will be a page or word limit imposed. Quite similar to the process used to consider a 
paper for publication in a journal, typically, conference papers are submitted anonymously 
(without the author[s]’ name[s]) and then distributed to a number of peer reviewers (often 
two or three) whose identities are unknown to the author(s). The peer reviewers provide 
feedback on the paper and recommend for or against its acceptance for the conference. It is 
also possible that the researcher submits a proposal or an extended abstract rather than a full 
paper for consideration for presentation at a conference. 

If accepted, there are usually two different presentation mechanisms. For a conference 
paper presentation, the paper will be presented orally by its author(s) alongside a small 
number of somewhat similar papers, with time provided for open question and answer and, 
sometimes, formal remarks from a scholar assigned as a respondent for the session. The oral 
presentation is often supplemented with a PowerPoint or Google slides display of key aspects 
of the paper and is typically timed (with 10—15 minutes as a typical length). For a confer- 
ence poster presentation, a hard copy, large-format visual display of the key features of the 
paper is made, and the author(s) stands in front of the display to field questions and engage 
in conversations about the research with poster session attendees (Figure 12.1). Sometimes, 
respondents are assigned to these sessions, as well, and therefore in addition to impromptu 
conversations with conference goers, more formal feedback might be communicated to the 
authors by the person in this role. Conferences may also convene virtually (as was the case 
during the COVID-19 pandemic, for instance) rather than face to face. 

Some research papers or summaries of research take the form of book chapters in an 
edited book, appearing alongside related chapters under a shared topic. If a scholar’s own 
research is quite extensive or, perhaps, has a number of related components, it may be suitable 
to take the form of an authored book. An authored book has a single author or co-authors 
who write the entire manuscript. Books are quite common in the humanities, and therefore 
an emphasis on this form can be found in the areas of the Communication discipline that 
draw from more humanistic intellectual traditions. Although not unheard of, a book-length 
treatment of quantitative research is rarer in published Communication scholarship compared 
to a book-length treatment of qualitative research. Indeed, most authored books that feature 
quantitative research do not emphasize a single, original research project but rather synthesize, 
critique, and review a vast body of research. 

For quantitative approaches to Communication, the journal article is typically considered 
the coin of the realm. Academic journals are published periodically online and/or in print 
and feature research and other forms of scholarship in article form. The nature and scope of 
quantitative research makes the journal article a good fit as a publication outlet, since journals 
vary in word or page limit but often allow about 25 pages of text (with additional references, 
tables, and figures) for each paper that they accept. 

A large and growing number of academic journals are devoted to or otherwise open to con- 
sidering Communication research for publication. There are also academic journals dedicated 
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Figure 12.1 Conference presentation 


Source: Wikimedia commons 


specifically to social justice research. Academic journals typically have statements of “aims 
and scope” that describe the sort of research they tend to focus on or favor for publication. 
They also have guidelines for authors to follow when submitting research. This information 
is found on the journals website and is important to read before deciding what journal to 
submit research to, as ideally there would be a close fit between the focus of the journal and 
the study itself. 

With a large number of possibilities, it can be challenging to figure out which journal to 
choose when submitting a study. In addition to reading information on journals’ websites such 
as aims and scope, four quick guidelines can help in the decision: 


e Ifyou think a particular journal might be a good fit, read some recent articles that 
were published in that journal. Do you think your study is on par with those recently 
published studies? Is your paper similar in some way? If so, those are good indications that 
this journal is, indeed, a good place to try. 

e Look at the studies included in the literature review and cited in the reference 
section of your paper. In what journals were those studies published? Perhaps one of 
those outlets would be a good fit for your paper, since the journal published one or more 
prior studies that were similar enough to warrant their inclusion in your literature review. 

e Read the list of editorial board members published on the journal’s website. 
From what you know or can find out about the editorial board members’ own research, 
do at least some of them do somewhat similar sorts of studies as the one you are submit- 
ting? Once again, if the answer is yes, that is another indication of a good fit. 


Communicating Research for Publication 327 


e Talk to your colleagues or, if you are a graduate student, your advisor, com- 
mittee members, and other professors about your paper and what journal they 
would recommend for it. Better yet, ask a trusted colleague to read your paper and then 
suggest one or more journals that might be a good fit. 


It is important to know that nearly all academic journals require that the papers submitted by 
authors for consideration for publication not be submitted simultaneously to any other jour- 
nal or publication outlet. Rather, academic journals tend to require exclusive consideration 
of a paper for publication. This means that once you have chosen a journal to submit a paper 
to, you must await the publication decision from that journal. 


Elements of a Journal Article 


As you learned in Chapter 1, there are a number of basic elements of a quantitative research 
paper that is written with the goal of publication in a journal. Of course, a research paper 
must have a title. Titles for academic papers should communicate clearly what the paper is 
about, but also be concise. The title should contain some of the keywords associated with 
the paper, in part so that if the paper gets published, it can pop up in online searches, such as 
through Google Scholar. A common convention that academics use in titling their papers is 
separating two clauses with a colon, the first clause perhaps more attention-getting and inter- 
esting and the second more informative and descriptive. 
Here are some examples of titles we have used in our own research: 


Saleem, M., Wojcieszak, M. E., Hawkins, I., Li, M., & Ramasubramanian, S. (2019). 
Social identity threats: How media and discrimination affect Muslim Americans’ identi- 
fication as Americans and trust in the U.S. government. Journal of Communication, 69(2), 
214-236. 

Saleem, M., & Ramasubramanian, S. (2019). Muslim Americans’ identity management 
strategies in response to media stereotypes and discrimination. Media Psychology, 22(3), 
373-393. 

Ramasubramanian, S., & Miles, C. (2018). Cross-national comparative framing of news 
stories about the Syrian refugee crisis: Arabic versus English newspapers. International 
Journal of Communication, 12, 4488—4506. 

Scharrer, E., & Blackburn, G. (2018). Cultivating conceptions of masculinity: Overall tel- 
evision use, genre viewing, and perceptions of norms regarding masculine gender roles. 
Mass Communication & Society, 21(2), 149-177. 

Blackburn, G., & Scharrer, E. (2018). Video game playing and beliefs about masculin- 
ity among male and female emerging adults. Sex Roles: A Journal of Research, 80(5/6), 
310-324. 

Sekarasih, L., Scharrer, E., Olson, C.J., Onut, G., & Lanthorn, K. (2019). Effectiveness of a 
school-based media literacy curriculum in encouraging critical attitudes about advertis- 
ing content and form among boys and girls. Journal of Advertising, 47, 362-377. 


The abstract appears at the beginning in published form but is often written last. The 
abstract is a summary of the main components of the research: its theory, methods, key find- 
ings, and importance. There is typically a rather tight word limit for the abstract imposed by 
the publication outlet. The task for the researcher, then, is to distill the research down to its 
most important parts. 
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Figure 12.2 Some quantitative research will be communicated in books or book chapters 


Source: Morten Oddvik, www.flickr.com/photos/mortsan/4570359817 


Here are two abstracts from our research to show as examples: 


This content analysis updates the literature regarding gender and the depiction of house- 
work in television commercials by exploring not just the distribution of chores but also 
the success or failure of chore performance. A sample of 477 commercials featuring 
domestic chores that aired in a week of primetime television programming on all of the 
broadcast networks was analyzed. Among the key findings: Male characters’ performance 
of chores was often humorously inept as measured by negative responses from others, lack 
of success, and unsatisfactory outcomes. These depictions have important implications for 
the learning of gender roles from television exposure. 

(From “Working hard or hardly working? Gender, 

humor, and the performance of domestic chores 

in television commercials” by Erica Scharrer, 

D. Daniel Kim, Ke-Ming Lin, & Zixu Liu, 

Mass Communication & Society, 2006) 


This paper uses survey methods (N = 385) to examine the underlying processes 
through which U.S. fans create meaningful relationships with Japanese media characters. 
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Specifically, this study tests an integrated model linking character perceptions, wishful 
identification, and parasocial relationships in the context of fans of shojo, a heroine- 
centric genre of Japanese anime. Results suggest that liking a heroine’s pro-social traits 
leads to greater wishful identification and more intense parasocial relationships with the 
heroine. Theoretical implications for reception studies and intercultural entertainment, 
and practical implications for role-modeling effects of positive female characters on young 
audiences are discussed. 

(From “Japanese anime heroines as role models for U.S. youth: 

wishful identification, parasocial interaction, and intercultural 

entertainment effects” by Srividya Ramasubramanian 

and Sarah Kornfield, Journal of International 

and Intercultural Communication, 2012) 


Following the abstract in a published piece, the author writes an introduction. The introduc- 
tion establishes the topic of the paper, often framed as an open question or a statement of the 
problem. An introduction should capture the interest of the reader, and therefore it should 
establish not just what the research is about but why the reader should care about the research. 
This latter issue—why the reader should care about the research—is associated with its social 
significance, its relevance to and importance for the world around us. Clearly, when con- 
ducting research with an interest in social justice, writing convincingly about why the reader 
should care about the study is a critically important step. 

As discussed in Chapter 1, the next element of a research paper is the literature review. The 
purpose of the literature review is to establish context for the study at hand by discussing 
relevant prior studies and concepts to establish how the new study fits in with what is already 
known. Research studies are similar to interlocking puzzle pieces, and in order to successfully 
place a new piece of the puzzle into a place where it fits well, you must know the layout of 
the closest surrounding pieces. 

One common strategy is to demonstrate through the writing of the literature review 
that the current state of knowledge on the topic is somewhat limited. Therefore, the study 
that you have undertaken in the paper is argued to fill an important gap, settle an unre- 
solved matter, establish a new context in which to test an existing theory, or otherwise 
address the limits of the knowledge to date. Literature reviews are often somewhat critical 
in nature, therefore, identifying both strengths and limitations of prior studies as well as 
strengths and limitations of the body of knowledge stemming from those prior studies as 
a whole. 

Literature reviews are typically organized by subsections, so that the existing research is 
divided up into logical, coherent subtopics that build together to establish the foundation for 
the study at hand. This helps communicate to the reader how various sub-topics and sub- 
themes can be brought together to establish the foundation for the study at hand. It is typical 
to end with the subsection that is most relevant to the study, as you can see reflected in the 
examples of labels used for subsections of the literature review that we have provided here 
from our sample studies. 

From the “Working hard or hardly working? Gender, humor, and the performance of 
domestic chores in television commercials” study: 


Social Statistics Regarding Gender Roles 
Gender Roles in Television Commercials 
Linking Humor and Gender Role Activity in Commercial Content 
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From the “Japanese anime heroines as role models for U.S. youth: wishful identification, 
parasocial interaction, and intercultural entertainment effects” study, in which two levels of 
subheadings are used: 


Japanese Animation and U.S. Audiences 
An Integrated Model of Wishful Identification and Parasocial Interaction 


Wishful Identification 
Parasocial Interaction 
The Hypothesized Model 


The next step in the sequence of elements of a quantitative research paper is the statement 
of the hypotheses and research questions (Figure 12.3). As you learned in Chapter 8, the 
hypotheses and, if applicable, the research questions come after the literature review because 
they should stem logically from the literature review. In other words, the predictions made 
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Figure 12.3 Parts of a research paper 
Source: BohatALA, https://bohatala.com/parts-of-a-research-paper/ 
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in the hypotheses should be based on the patterns found in prior studies or prior tests of the 
theory. Those patterns will have been communicated in the literature review. 

Following the statement of the hypotheses and research questions, the methods section 
appears next. Clarity and comprehensiveness are key in this section so that the reader can 
know exactly how (and why) each step of the research process was carried out. The concepts 
at play in the literature review should be the same or very similar to the concepts at play in 
the methods section. 

Some components of methods sections are common to quantitative research papers regard- 
less of the specific method (experiment, survey, content analysis, etc.) employed. Methods 
sections for quantitative research papers typically have a subsection about the sample, partic- 
ipants, and/or the sampling process. Many of the aspects of Chapter 4 in this book will 
be reflected in the write-up of this subsection. For a content analysis, for instance, you would 
describe how and why you chose the particular texts that comprise the sample and how many 
texts you wound up with for the analysis (sample size). For both experiments and surveys, the 
sample will consist of the research participants, and in this subsection of the methods section 
you would explain how and why you invited particular people to participate as well as how 
many agreed and did, indeed, participate (sample size, recruitment, and response rate). 

Methods sections for quantitative research papers also typically have a subsection on meas- 
ures, in which you explain how concepts were operationalized (recalling the lessons of Chap- 
ter 3). For content analyses, this is where you would explain the coding scheme in detail. As 
you may recall, the coding scheme outlines the conceptualization and operationalization of 
the variables measured in the study. For surveys, the measures are the items used on the ques- 
tionnaire that respondents filled out, and in this subsection you would explain from where 
those measures were derived (Were they used in prior research? By whom? Or did you create 
them for your own purposes? What principles guided their creation?) and why they are reli- 
able and valid measures of the underlying concepts at play in the research. For experiments, as 
you learned in Chapter 6, the measures may take the form of self-report items on a question- 
naire (similar to surveys) or may be observations of some kind. In either case, the measures 
must be fully explained in this subsection of methods, and it should be clear why the use of 
those measures, in particular, is an effective way to measure the concepts being tested in the 
research. 

Research that involves human subjects will also have a statement in the methods section 
about the protocol used to establish research procedures associated with the protection of 
human subjects. For instance, most academic journals require that researchers state whether 
the research has undergone review by the Institutional Review Board and has been approved, 
sometimes even noting the date on which the approval was received. Somewhat relatedly, 
also for any research involving human subjects, the methods section should contain informa- 
tion about what the participants were told about the study when they decided whether to 
participate or not. 

Methods sections for quantitative research studies also might have a data analysis plan, 
sometimes (but not always) labeled as such using a subheading. In this section, the plan for 
what statistical procedures and tests will be conducted to test the hypotheses and explore the 
research questions are communicated to the reader. Sometimes the software programs (like 
SPSS) used to perform those statistical procedures and tests will be noted here, as well. 

Other components of methods sections are likely to be unique to the specific research 
method employed. 


e For content analyses, there should be a subsection on intercoder reliability that reports 
how and for how long coders were trained to do the coding, the number of coders, any 
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salient aspects of the coders themselves (like whether they were also authors on the study 
or, perhaps some aspect of their identities such as their gender), how intercoder reliability 
was calculated (from what portion of the sample and how that portion of the sample was 
chosen, what formula was used), and the evidence for the degree to which intercoder 
reliability was established for each variable. 

e For experiments, there should be a subsection on the procedures used in the study, 
including what stimulus or stimuli were used (and why); the order in which research 
participants were asked to do particular tasks (like fill out a pre-test questionnaire or be 
given the stimulus followed by the administration of a questionnaire); the results of a 
manipulation check, if relevant (in which participants are asked, after the fact, whether 
they did find the stimulus to have the properties the researcher assumed it had); and 
whether and how participants were debriefed (meaning that they were told the purpose 
of the research, after the fact). 


The next section in the typical organization of a quantitative research paper is the results 
section, and this is where the statistical findings are communicated. It is common practice to 
begin this section with descriptive statistics—often labeled as such—to detail the means and 
standard deviations or the frequencies associated with each main variable. Any data reduction 
or transformation is typically outlined next (and, again, is often organized under a subhead- 
ing), followed by the main statistical analyses. When reporting the main statistical analyses, it 
is important to include all of the components for each type of statistical test that we outlined 
in Chapter 10. This typically entails the result for the test statistic itself, the degrees of free- 
dom associated with it (when relevant), and—importantly—the associated probability level. 
It is also vitally important to link results back to hypotheses and research questions, making 
it abundantly clear in the write-up whether the findings allow hypotheses to be supported or 
rejected or what specific answers they provide to each of the research questions posed. 

When writing the results section of a research paper, you will most likely also create tables 
and/or figures to visually display key findings. Tables are a way to place the most important 
numerical results from statistical tests in a well labeled and easy-to-read display using rows 
and columns. Figures do not just contain numbers (as tables do) but rather illustrate findings 
through graphs, maps, charts, or other images. Tables and figures are used to supplement the 
text-based write-up of results. Therefore, in the place in the results section where you are 
discussing particular findings that also appear in a table or figure, you would refer the reader 
to the table or figure in question, typically through the use of the phrase “see Table 1” or “see 
Figure 1.’ Indeed, the tables appearing in the paper are numbered sequentially (from Table 1 
to Table 3, for instance, if you have three tables) and should be discussed in that same order 
in the paper. (In other words, in results, do not start with a discussion of Table 3. Start with 
Table 1.) Figures are treated in exactly the same manner and are typically numbered sepa- 
rately from tables. Although tables and figures are discussed in the text using phrases like “see 
Table 1,” it is standard practice for the tables and figures themselves to appear at the end of the 
paper, after the references, with each table or figure on its own page. 

In fact, there are standard formats for reporting the relevant parts of the statistical tests that 
we introduced in Chapter 9 and for crafting tables and figures. Many journals that publish 
quantitative Communication research require that APA style guidelines are followed, and 
therefore that is a good resource to consult. Examining the format and contents of tables and 
figures in prior published work can also be a big help in guiding your own creation of these 
key features of a quantitative study. 

The discussion section is the last major section of a quantitative research paper. In this 
section, you step back from the specificity of the quantitative results and interpret the main 
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patterns and findings using more of a “big picture” lens. You explain in words rather than 
with numbers and statistical tests what the most important results are, what they mean, and 
why the reader should care about them. A discussion section typically has the following 
components: 


e Summary of major results: Typically, discussion sections begin with a brief account of the 
most important findings from the study. Once again, when you write about the findings 
this time, you do so without the specific numbers associated with those findings, focus- 
ing, instead, on the overall patterns gleaned from the specific numbers. 

e Interpretation of major results: Woven together with a statement of the most important find- 
ings is interpretation of those findings, answering the question: What does it all mean? In 
interpreting findings, oftentimes you will cautiously speculate about the implications of 
the results, using phrases like “This appears to suggest that” or “This may have occurred 
because.” In fact, if hypotheses were not supported by the data, it is customary to specu- 
late about why that may have been the case in the discussion section. If hypotheses were 
supported, speculation may also be called for, as you explain what such support means for 
the extension of theory and research in the topic area. 

e ~~ Connection back to literature review and theory: Particular former studies from the literature 
review should be referenced in the discussion section once again, but this time in direct 
connection with the findings of the current study. If the current findings align with find- 
ings from past research, you discuss that here. If the current findings conflict with former 
findings, you discuss that here, as well, and you speculate as to why that might be the case. 
Sometimes, a small number of entirely new former studies from the body of literature 
(i.e., those that were not in the literature review) are brought in to this part of a discussion 
section, as well, if they turn out to be relevant in light of the findings. 

e Discussion of real-world implications: Given that social justice research, in particular, strives to 
be action-oriented, it is also important to establish what the findings mean for the work 
of organizations, the formation of policies, the practices of companies or communication 
industries, and other real-world considerations. 

e Identification of limitations of the study and call for future research: Every study is limited in 
some way, and it is always best to be straightforward and self-aware about those limita- 
tions, discussing them in this component of the discussion section. Often, the limitations 
are a good way to transition into calling for future research. In calls for future research, 
the researcher suggests what subsequent studies could or should do in order to follow up 
on the current study, addressing its shortcomings or extending knowledge further in light 
of what was found. 

¢ Drawing conclusions: Final statement of strengths/contributions of the study at hand, and final 
statement of its importance: In addition to limitations, every study also has its strengths, and 
it is good practice to end the write-up of your paper with a reminder to the reader of the 
contributions that the study has made (despite its flaws). It is also a good idea to empha- 
size once again why the topic is worth pursuing in ongoing research. A study ends with 
conclusions of this sort, which can be thought of as key take-away messages. 


After the discussion section, the references appear, beginning on a new page. The references 
are organized according to an accepted style guide such as APA, Chicago, or Modern Lan- 
guage Association (MLA). As we mentioned, most quantitative research in Communication 
uses APA style, and that is the style used in this book, as well. It is important to make sure you 
are using the most recent version of the style guide and applying it consistently, not just to the 
reference section but to all components of the written research paper. 
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Peer Review 


The chief reason why published research is generally regarded as more trustworthy than 
unpublished research is because published research has undergone peer review (Nicholas 
et al., 2015). Peer review is the process by which a research paper is assigned (by the editor of 
the academic journal, for instance) to be carefully assessed by two, three, or more experts in 
the field. The peer review process is typically a double-blind review: The identities of both 
the authors and the expert reviewers are unknown to the other. The expert reviewers provide 
detailed feedback on the strengths and limitations of the research and on the paper itself and 
decide whether to recommend that it be: 


e Accepted for publication 
e Revised and resubmitted for continued evaluation 
« Rejected 


Alas, the third possibility—rejection—is the most common response. Academic journals 
receive many more submitted research papers than they are able to publish. In fact, some of 
the most selective journals in the Communication field may have acceptance rates of around 
10%. Part of surviving and thriving as a researcher involves being able to deal with rejection 
and cultivating the resilience that it takes to keep trying to get one’s work published even 
against such daunting odds. 

The second possibility, receiving a “revise and resubmit” invitation to make changes to the 
paper in light of the detailed feedback received and submit it for continued consideration for 
publication, does not guarantee that the paper will be accepted. The author may receive a set 


Figure 12.4 Pro-science protest 


Source: Takver from Australia, www.flickr.com/photos/takver/33824095770/ 
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of detailed reviews, make revisions and resubmit, and still possibly have the paper be rejected 
or receive another set of reviews to respond to, as well. Even after two (or more!) rounds of 
revision, it is possible (although, thankfully, relatively unlikely) that the piece will be rejected. 
Yet, a revise and resubmit decision allows for continued opportunity for improvement to the 
paper which, ideally, gets the author closer to a favorable publication decision. 

Despite the merits of peer review, there are certainly drawbacks, as well. Reviews are not 
always helpful. Sometimes they are off base or they contain erroneous assumptions. Some- 
times they are unduly harsh. Sometimes they are too vague to be of much value in improving 
the paper. Sometimes, reviews received from two or more anonymous reviewers conflict with 
each other. Sometimes reviewers display bias against the research topic, the research method, 
the theory, or some other aspect of the paper. And many times the peer review process takes 
quite a while to unfold, and the author is anxious for an outcome. This time-consuming 
process can be especially frustrating for researchers working with community partners or on 
timely real-world issues, as is often the case with researchers committed to social justice. 


Limitations of the Publication Landscape 


It is important to know that some librarians, researchers, and scholars are pushing back against 
the structures of the academic publishing industry. Currently, a small number of publishers 
own a majority of the journals in which academic research is published. Elsevier, the largest 
publisher of academic journals, for instance, is responsible for the publication of some 3,000 
journals, accounting for about 18% of all research published across the world (Resnick & 
Belluz, 2019). A 2014 report published by the National Communication Association showed 
that the majority (61%) of Communication journals are published by either Sage or Taylor & 
Francis, also fairly large publication houses. NCA, ICA (again, International Communication 
Association), and AEJMC (Association for Education in Journalism and Mass Communica- 
tion) sponsor their own journals, as well, sometimes in partnership with a publishing house. 

Published articles often appear behind a paywall, which means that in order for most 
users—including members of the general public—to access the articles, they must pay a 
fee. This is a limitation especially relevant to social justice research, since unequal access to 
resources is a fundamental justice-related principle. University libraries also pay potentially 
quite large subscription fees to publishers to allow their users to access many published 
articles without the users themselves having to pay. In just a single example, for instance, the 
University of Virginia paid Elsevier $1.83 million for its journals in 2018 (Resnick & Belluz, 
2019). As of February, 2020, the University of California system has dropped its subscription 
to Elsevier journals, a subscription that brought a price tag of about $11 million annually 
(Resnick & Belluz, 2019). 

It may surprise you to know that authors do not receive any money for having their research 
published in academic journals, even though it is their work being published. And, indeed, 
the peer review process discussed earlier is also a service that academics typically provide for 
free. Why, then, would an academic researcher continue pursuing the publication of their 
research in academic journals owned by the major publishing companies? Mostly because the 
major publishing companies own many of the more prestigious journals, and many academics 
operate within university structures and cultures that have incentives associated with having 
their work appear in these outlets. 

Given this context, it is perhaps unsurprising to discover that there has been a growing 
movement to chip away at the dominance of the major publishing companies. The open 
science movement has paved the way for the growth of open access journals, those freely 
available to users. There is a wide array of open access journals, and the variation in their 
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approaches to publishing can be difficult for researchers to negotiate. Although many do use 
the peer review process that is largely considered the gold standard in publishing, some do 
not, and therefore there may be a concern that research appearing in the latter is not fully 
vetted. Some open access journals are published online only, sidestepping the costs for pub- 
lishers associated with print-based publishing. In the early days of online journal publishing, 
there was widespread bias against such journals, with many assuming online journals were 
not as reputable as print-based journals. Some of that bias may linger today. Still other open 
access journals offer a hybrid model, in which authors can pay a fee (covered either by the 
authors themselves or by their institution) to have their work be published as open access. 
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Figure 12.5 Cartoon showing the ‘publish or perish' culture of academia 


Source: CartoonStock.com 
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Researchers tend to be wary of paying for any aspect of the publication process, partly because 
they fear doing so will erode the rigors of peer review (Nicholas et al., 2015). For your infor- 
mation, the International Association of Media and Communication Research (IAMCR) 
keeps a list of open-access journals in the field on its website, as well: https://iamcr.org/ 
open-access-journals. 

Standard publication agreements used by academic journals usually do not allow the final, 
published version of articles to be shared freely but often do permit a less formal pre-publica- 
tion version to be posted online to allow wide and free access. Thus, some researchers are also 
making prepublication versions of their academic papers available for users to access for free, 
as well. In fact, online platforms such as ResearchGate or Academia.edu are sites on which 
research can be shared, as long as doing so does not violate publication agreements. University 
libraries may often host prepublication versions of researchers’ studies on their websites, too. 

Finally, there are concerns, as well, about the culture created by a system that pushes 
researchers to be published and/or to be published in particular journals in order to advance 
in academia. One possibility is that such a culture creates the temptation for researchers to 
fudge their data, make up their findings, or otherwise engage in unethical research prac- 
tices. Reliance on the current system of considering publication in highly selective journals 
as the most important way to gain status in academia has been implicated in a number of high 
profile cases of data fabrication or data falsification, such as changing results to make them 
appear statistically significant or supportive of hypotheses (called p-hacking), or otherwise try- 
ing to game the publication system. We discussed the possibility of pre-registering hypotheses in 
Chapter 8, which is one safeguard against questionable research practices such as these. 


Writing for the Wider Public 


Writing for publication can be a necessary condition as well as a rewarding experience for 
many researchers. We would argue, however, that it is not sufficient for those striving to have 
their work matter for the greater good. Given how difficult it is to get research published, 
it is tempting to have publication signal the end of a research project. Yet especially if a 
researcher has already secured a stable position or otherwise has the freedom to think outside 
of the “publish or perish” system, a great deal of good can be gained through wider efforts of 
research dissemination, community dialogue, and social action. We agree with health systems 
researchers Gavin Bennett and Nasreen Jessani (2011) who state, “Successful publication on 
the part of research is not ‘job finished’ It is job started’. . . to drive the research findings 


Figure 12.6 Some researchers use social media to engage with a wider public 


Source. MKH Marketing, https://commons.wikimedia.org/wiki/File:Social_media_class.jpg 


338 Communicating Research for Publication 


to wherever they need to be to provide real and maximum benefit to policy, to practice, to 
people” (p. xxii). 

A very important goal of social justice research is to break the boundaries associated with 
“the ivory tower” of academia. Social justice research often has a goal of co-producing knowl- 
edge with community partners and/or bringing knowledge back to the people. Knowledge 
is not just produced for the purpose of the researcher’s advancement and filed away; it is put 
to use. There are many tools available to meet these goals. In this chapter, we will focus on a 
short selection. 

Blogs are usually informal and conversational in tone, making them a good way to distill, 
describe, and discuss research evidence and commentary in an easy-to-follow and engaging 
manner. Blog posts by academics are often evidence-based, which may set them apart from 
purely opinion-based writing. Most blogs are interactive, meaning that readers of the blog can 
post comments and the researcher can comment on those comments, allowing for dialogue. 
This is important in that it reminds us that, from a social justice perspective, “writing for the 
public” should ideally be a conversation rather than a one-way process in which the researcher 
simply imparts wisdom or research-based insights. 

Blog posts should have attention-getting titles, should be relatively short (often 800 words 
or so), may feature visuals, and—although they can vary in tone from more to less formal— 
should generally avoid the sort of dense and impersonal writing that marks a great deal of 
academic work (Mollett, Brumley, Gilson, & Williams, 2017). Blogs that have attracted many 
users may also capture the attention of journalists at more traditional news organizations, 
opening up the opportunity to reach public audiences even further. The researcher posting 
the blog can tag the journal in which the article was published or can share the post on social 


Figure 12.7 Writing for the public 
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media. Platforms available to create blogs include WordPress, Blogger, and Medium. On the 
downside, blogs do take effort to create as well as sustain. 

Here is a short list of blogs and/or organizations that host blogs that may be relevant to 
those studying Journalism, Communication, and Media Studies: 


e PRESSTHINK by New York University Professor Jay Rosen 

e Journalism Research News 

e Online Journalism Blog 

e Poynter Institute 

e Knight Center for Journalism in America at University of Texas Austin 

e How Fantasy Becomes Reality by Fielding Graduate University Professor Karen 
Dill-Shackleford 

e Organizational Communication Research Center blog from the Institute for Public 
Relations 

e The Oxford Internet Institute 

e The Centers for Disease Control’s blog for Health Communication 

¢ The Director’s blog of the Health Communication Center at the University of Texas at 
Austin’s Moody College of Communication 

e The Campaign Workshop Blog 

e The Project on Ethics in Political Communication 

e The Center for Political Studies, a joint project from the University of Michigan and the 
Institute for Social Research 

e The London School of Economics, Media@LSE 

e Media Shift 

e GLAAD 

e Name It. Change It., from She Should Run and the Women’s Media Center 

e Adbusters 

e Muslimah Media Watch 

e Speak Schmeak 


And there are many blogs about academic writing and or thriving in academia, including the 
Academic Writing Blog (through the Academic Writing Success organization), The Research 
Whisperer, From PhD to Life, PLOS ECR (for early career researcher), and Get a Life PhD. 
Inside Higher Ed and the Chronicle of Higher Education also both feature blogs. 

Social media, including Twitter, Facebook, YouTube, LinkedIn, and Instagram, are also 
used by researchers to inform audiences of new research and/or to comment on issues and 
topics of the day. Incorporating photos or videos can help such posts gain attention and can 
illustrate key points or themes. Hashtags can be used in some platforms to identify key words 
in the content of the post, which can help make those posts more prominent in searches. 
A researcher can participate in social media communities associated with the professional 
organizations in Communication such as NCA and ICA. 

Websites are also useful tools for researchers to post descriptions of their area(s) of exper- 
tise, information on what they are currently working on, and recently published research 
(perhaps in pre-print form). Linking to journals, conference websites, or other web locations 
of interest is also a good idea. Blog posts can be uploaded here, as well, and those might 
recount recent invited lectures, participation in on-campus events, or other activities. Com- 
pared to blogs, websites tend to be updated less regularly (although they certainly should still 
be updated), to adopt a one-way dissemination of information (as opposed to something more 
interactive), and often are more formal (Mollett et al., 2017). Websites also tend to cost more 
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than blogs to create and maintain. Many universities allow faculty members to host their per- 
sonal websites on the official pages associated with the university or department. Please see 
www.drsrivi.com for Prof. Ramasubramanian’s website. 

Despite the advantages of using these internet-based tools, there are drawbacks, as well. The 
main limitations researchers identified with using social media, in particular, to disseminate 
research findings were that (1) such a practice is not typically valued within the institutional 
culture of higher education, (2) there is no equivalent to peer review that can act as a means 
of determining the quality of what is being disseminated, and (3) some see more potential 
threats to their reputations from the use of social media than benefits (Nicholas et al., 2015). 

Indeed, researchers may be wary of feeling obligated to engage with broad publics on top 
of all of the other expectations and requirements associated with their careers (Mollett et al., 
2017). They wisely ask whether such efforts are valued in their employer’s reward structures. 
In other words, if a researcher is trying to get tenure and/or be recognized for meritorious 
performance, will a blog post count toward their case? How much will it count relative to 
other (perhaps more traditional) scholarly forms? These are important questions to ask, and 
the answers to these questions are likely to vary widely across different contexts or at different 
stages of a researcher’s career. 

It is also important to acknowledge that some academics have been penalized by their 
institutions for comments they have made on social media. Still others have received racist, 
sexist, or other hateful comments on social media platforms (Carrigan, 2016). Finally, some 
academics are also uncomfortable with the goal of public engagement because they associate 
such efforts with self-promotion. In fact, some academics judge those who engage broadly 
with the public as somehow inferior or less “pure” in motives. On the other hand, William 
Tyson (2010), author of Pitch Perfect: Communicating with Traditional and Social Media for Schol- 
ars, Researchers, and Academic Leaders, suggests that researchers think of public engagement as 
less about promoting themselves and more about bringing to light the ideas and issues at stake 
in the research. With such a view, engaging with the public is not “shameless self-promotion” 
but rather is about the collective good. 


Scholar Spotlight: Dr. Karen Dill-Shackleford 


Karen Dill-Shackleford, Professor of Media Psychology, Fielding Graduate University, 
studies the potential for media storytelling to immerse audiences and otherwise shape 
their daily lives. The author of the books, How Fantasy Becomes Reality and Finding Truth 
in Fiction (with Cynthia Vinney), the editor of the Oxford Handbook of Media Psychology 
and Real Characters, and the author of many journal articles, Dr. Dill-Shackleford is also 
committed to public engagement as well as bringing research to bear in policy making. 
She gave expert testimony before Congress on the topic of media violence in 2000 
and on misogyny in rap music lyrics in 2007. She currently writes two blogs, one for 
Psychology Today and one for Oxford University Press as part of Academic Insights on 
the Thinking World. She has also written for health-care practitioners in the journal, 
Pediatrics, and has recorded a TEDx Talk called Sherlock Holmes and Harry Potter are Real 
(www.tinyurl.com/SherlockHarryPotter). 


Q: In your career, you often find ways to bring ideas and evidence from research to either policy 
making contexts or to forms (like blogs or a TEDx Talk) where you have the potential 
to reach a general audience. Why have you prioritized this sort of work? 
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Figure 12.8 Scholar Spotlight: Dr. Karen Dill-Shackleford 


Dr. Dill-Shackleford: That’s a great question because academics are often rewarded for 
publishing in scientific journals more than in forums that reach a more general 
audience. I have indeed prioritized communicating science in the public interest 
even knowing that it takes time away from doing and publishing research. I do it 
because I think the research that my colleagues and I do can actually make a differ- 
ence in people’s everyday lives. I know that I reach a wider and more diverse audi- 
ence through these types of forums. When I write a blog entry or give a talk for a 
general audience, I’m thinking about that one person who might learn something 
that’s useful to them that they might not have heard otherwise. Is reaching that one 
person important? Absolutely; it’s everything. 

Q: In your experience providing expert testimony to Congress, have you found that quantita- 
tive research evidence, in particular, has an important role to play? Why? How have policy 
makers responded to this sort of evidence? 

Dr. Dill-Shackleford: Scientists strive to answer the burning questions in their field 
with hard evidence. We take an educated guess, but then we gather data to find 
out if there’s really reason to believe what we guessed to be true. I’ve found that 
there’s a general acceptance of the idea that evidence is important. Experimental 
evidence is valued because it’s the main method we have of finding out what 
causes something to happen. I do think this kind of quantitative data is highly 
valued even outside of science. I have been surprised to learn that non-scientists 
do not necessarily interpret quantitative results in the same way that scientists do. 
For example, the first time I testified before Congress, the American Psycho- 
logical Association cautioned me that members of Congress do not necessarily 
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understand the data the same way that scientists do. In that case, I had to figure 
out how to explain it so that non-scientists would understand what was most 
important. 

Q: What are the opportunities and drawbacks for researchers who want to get their research and 
the research of others into the hands of change makers, such as practitioners, policy makers, 
or non-profits? 

Dr. Dill-Shackleford: One of the challenges is what I explained earlier: that you must real- 
ize that your work needs to be translated for the audience you are speaking to. You 
also work to make that message engaging. This need for science to be communicated 
in a clear and engaging way is why the Alan Alda Center for Communicating Sci- 
ence was started at Stony Brook University (see aldacenter.org). Though commu- 
nicating science in the public interest can be a challenge, it also has the potential to 
be some of the most rewarding work you can do. For example, knowing my work 
would be used by organizations like the Kettering Foundation, the U.S. Congress, 
or by pediatricians (via the journal Pediatrics) gives me a sense that I am sharing good 
ideas from science with the people who can make practical use of them. 


Although we do not have the space to discuss more visual rather than text-based com- 
municative forms, it is also important to acknowledge that documentary film, theater, digital 
storytelling, photo exhibitions, and other visual communicative forms are frequently used in 
social justice research contexts, as well. As a follow up to Chapter 11, too, we should point 
out that sharing research findings with community participants, community partners, and 
other stakeholders through such means as offering town hall sessions or making guest visits to 
organization’s meetings are best practices. 


Engaging with the News Media 


Researchers can be among the expert sources called upon to comment on current topics and 
events in the news. Occasionally, too, a researcher’s own study may be deemed newsworthy, 
and journalists may seek out a researcher to discuss that study. Researchers appearing in the 
news have an excellent opportunity to reach a wide public audience. They can provide inter- 
pretations and discussion of timely and socially significant issues and show the public how 
research (theirs or others’) informs those issues. 

Tyson (2010) has found that journalists often conduct Google searches for scholars who 
might serve as news sources, in addition to reading the content of other news and informa- 
tion outlets, and asking other experts for recommendations. When they reach out to scholars, 
journalists are looking for someone (1) who will respond promptly, (2) who is clear about 
what they have to say, and (3) who can speak specifically on the matter the journalist is writing 
about. Longer-term relationships form, as well, when researchers become known to journal- 
ists as reliable sources of good insights. 

Researchers being interviewed as news sources should follow a number of guidelines. They 
should refrain from providing lengthy, drawn out answers to journalists’ questions and, instead, 
should get right to the point. They should be as engaging as possible. They should do their 
best to answer the question at hand but be sure, also, to use journalists’ questions as oppor- 
tunities to focus on what they themselves want to say. In other words, they can direct the 
conversation with the journalist to a point they want to make, a research pattern they want 
to emphasize, or a recommendation that they think is justified based on the evidence. They 
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should avoid jargon, as well as avoid focusing on too many of the more technical details 
related to research. They should always be truthful, even if that means answering some jour- 
nalists’ questions with “I don’t know,” or turning down an interview request if they do not 
feel confident on the subject. 

To assist with the translation of Communication research into more easily digestible, “bite 
size” summaries, researchers at Radboud University in the Netherlands developed a website 
called Bitescience (www.bitescience.com). Journalists (and others) can read succinct, clearly 
written summaries of research findings on a number of topics on the site. The Bitescience 
mission statement reads: 


Communication and media researchers are continuously gathering valuable insights 
on how individuals, organizations, and societies use, create and are affected by com- 
munication through traditional and new media. These insights are highly relevant for 
communication professionals, policy makers, and the general public. Unfortunately, 
academic research is often not accessible to nonacademic audiences. Scientific reports 
are not only difficult to find and get access to; they are often too complex and, let’s 
face it: a bore to read. Therefore, the mission of Bitescience is to provide access to the 
vast body of scientific expertise about communication and media and to present it in 
an attractive manner. 


Another option for researchers to reach the general public is to write opinion pieces or 
editorials (also known as op-eds) for news outlets. Once again, to increase the odds that an 
editorial will be chosen for publication, the researcher should write it in a clear, engaging style 
and should connect research evidence with current controversies, issues, topics, or events. For 
any type of news coverage, then, often researchers should think about a hook, a way to connect 
the research to a current event, issue, topic, controversy, or concern. 

The Conversation is an online publication that features “crossover” writing by academics 
providing evidence-based commentary on current and ongoing issues to a broad audience. 
Founded in Australia in 2011 and now appearing in eight countries, The Conversation’s tagline 
“Academic rigor. Journalistic flair.’ communicates well the approach taken by the editorial 
staff. Academics pitch their ideas for such writing via The Conversation’s website, https:// 
theconversation.com/us. After a successful pitch, the academic writer works with an editor 
to ensure the article is engaging and readable. Articles appearing in The Conversation can also 
be picked up by other news outlets, further expanding reach. The first author of this book 
published an essay, titled Why are sitcom fathers still so inept?, just before Father’s Day in 2020, 
for instance, and it garnered over 90,000 readers on The Conversation site itself or in one of the 
outlets in which it was republished (Figure 12.9). 

Despite the potential benefits of such opportunities to reach a broad public, there are 
challenges, as well. One much-discussed issue is the difficulty of reconciling the built-in 
uncertainty of scientific research with the more unambiguous answers that journalists may 
seek. In the book, The Public Professor: How to Use Your Research to Change the World, econo- 
mist M.V. Lee Badgett (2015) cautions researchers communicating to the general public to 
be aware of the tendency to add caveats to just about everything they say. Badgett writes, 
“The need to hedge comes naturally to us because of our professional scientific process and 
standards: We can never know anything with absolute certainty. . . . In the larger world, 
though, I think it’s safe to say that the caveats that academics use drive the media and most 
other nonacademics crazy” (p. 121). Badgett argues for a position in the middle ground, 
keeping only the most important caveats in place as we speak about research to journalists 
and/or the wider public and being bold enough to draw conclusions while still acknowl- 
edging complexities. 
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From Homer Simpson to Phil Dunphy. sitcom dads have long 


been known for being bumbling and inept. 
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ha ~ at to be serious, calm and wise, if a bit detached. In a hs 
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act Th mas Disclosure statement 


Figure 12.9 An essay published in “The Conversation” 


Source: Stephen Warren 


Some studies have found that journalists do tend to present research evidence in relatively 
unambiguous terms, leaving out the disclaimers that scientists often use to discuss research 
(Brechman, Lee, & Cappella, 2009; Jensen, 2008). Journalists may do so because they want 
to ensure that their news stories are easily understood by members of the public (Ebeling, 
2008). The onus falls on the researcher, then, to explain even the uncertainties and the 
complexities of the research evidence clearly and plainly. One way to ensure that one is 
doing so is to avoid the use of jargon or other language that is difficult to understand. 
Academics may be so accustomed to using certain highly specialized words and phrases that 
they may lose sight of the possibility that non-specialists have no idea what those words and 
phrases mean. 

Researchers may fear that they will be misquoted, quoted out of context, or otherwise 
not represented well in the final news story. Somewhat relatedly, academics or other research- 
ers may be surprised to find that a rather lengthy conversation they had with a journalist has 
been reduced down to only one or two short quotes. Sometimes, in fact, journalists inter- 
view a researcher as a source and then decide not to use the information gleaned from that 
interview at all. These possibilities are, for the most part, out of the control of the researcher. 
All the researcher can do is try to be a good source—responding clearly and accurately with 
engaging answers to journalists’ questions—and then let the chips fall as they may. 

One very practical potential difference between journalists and researchers is their sense 
of time and their response to deadlines. Often, when journalists reach out to researchers 
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as potential news sources, those journalists need a reply within a short timeframe, since they 
are working on deadlines to produce their content. Researchers, on the other hand, may find 
those inquiries from journalists occur at inopportune times, may not be as inclined to respond 
immediately, and may be more used to a slower deadline schedule. 

Finally, in terms of advice to researchers for engaging with journalists and attempting to 
get news coverage of their new research findings, you should know that most colleges and 
universities have news office staff members who can help. If you are affiliated with a college 
or university, get to know the staff at the news office, share your most important new find- 
ings with them, think about ways that your research fits with events and topics in the news, 
and reach out to news office staff then. They can help you write a press release, or better yet, 
write one for you. A press release is a succinct, attention-getting written announcement 
of something newsworthy that is sent to journalists to inform them. There are also resources 
available online to assist a researcher in writing a press release. Staff can also help you prepare 
for an interview, contact news outlets, and perhaps even put you in touch with journalists 
that they know. 

Inside Higher Ed reporter Robert Kelchen (2018) issued these five media engagement tips 
for academics that serve as good take-aways from this section: 


1. “Work on being comfortable speaking to broad audiences. 

2. Make yourself known to members of the media. 

3. Be available on fairly short notice whenever possible. 

4. Think about other potential sources who can also talk about the topic. 
5. Realize that not every interview ends up with your name in lights.” 


Writing to Shape Policy 


Public policy is an important arena in which knowledge garnered through Communication 
scholarship can be put to good use. Broadly defined as the development of laws, rules, recom- 
mendations, and courses of action, as well as the principles that underlie them, policies shape 
daily life and society as a whole. By doing research to shape or to respond to policy, research- 
ers committed to social justice and/or public action can make a direct impact. Such efforts 
are also another means of making sure that research and knowledge are public resources, 
benefiting the people. 

Governmental and legislative processes are major aspects of public policy formation, and, 
indeed, most of our discussion in this chapter will be about researchers engaging with that 
type of policy. The Federal Communications Commission, the Federal Trade Commission, 
the Surgeon General’s Office, and Congress are examples of U.S. governmental bodies whose 
work is or has been relevant to Communication research. As we will show in this chapter, 
Communication researchers (1) have been called upon to help shape the work of these bodies 
while members of those bodies are in the process of making their decisions and (2) have stud- 
ied the impact of the policies formed by these bodies after that they have been implemented. 

Lavis and colleagues (Lavis, Lomas, Hamid, & Sewankambo, 2006) have identified a use- 
ful way of thinking about the possibilities available to researchers to participate in policy 
processes: 


Push: The researcher pushes out information pertaining to their research to policy makers 
or other research users, presenting it in forms those policy makers/users can readily access. 

Pull: The policy maker/research user seeks out the researcher, asking for evidence or 
information needed for particular initiatives. 
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Exchange: Policy makers/research users and researchers form reciprocal, mutually ben- 
eficial relationships. 

Integrated: Partnerships between researchers and policy makers/research users are sup- 
ported through wider systems, such as institutions that broker and help sustain such 
partnerships. 


An example of an integrated approach is Research4Impact, an initiative co-founded by 
Cornell University Professor of Government Adam Seth Levine, Columbia University Polit- 
ical Science Professor Donald P. Green, and University of Illinois at Urbana-Champaign 
Political Science and Statistics Professor Jake Bowers. Research4Impact engages in “match- 
making” across social scientists, policy makers, and practitioners, helping establish and main- 
tain relationships between and among individuals across the divides of theory and practice 
and supporting such partnerships through workshops and information sharing. Their vision, 
“A world in which people with diverse forms of expertise seamlessly learn from each other, 
particularly those working to understand and solve complex social problems” (www.r4impact. 
org/welcome), expresses well why such partnerships can have an impact. 

Building relationships between researchers and policy makers has been identified as the 
single most important predictor of the ability of research to inform action through policy 
(Innvaer, Vist, Trommald, & Oxman, 2002). The degree to which research is timely and rele- 
vant is also a vitally important predictor of success in making research meaningful in the realm 
of policy. Relevance pertains to the alignment between the nature and topic of the research 
and the needs and priorities of the policy maker. Timeliness often calls for a “rapid response” 
on behalf of the researcher, understanding that policy makers have their own timelines and 
perhaps more immediate needs. Having the right evidence at the right time for policy mak- 
ing may seem difficult to achieve, but if ongoing relationships with policy makers are formed, 
researchers’ understanding of these moments will be facilitated. 


Forms and Guidelines for Policy-Related Writing 


Writing for the policy making process can take a number of forms. One pagers are ideal 
for busy policy makers, as they establish the issue clearly and succinctly and list a small num- 
ber of key take-away messages associated with the issue. Executive summaries (typically 
around three pages) provide more detail and discussion of the issue, using brief summaries and 
descriptions of research evidence to show why action is needed and/or why we might expect 
a particular action to have a particular outcome. Full papers or reports (usually around 20 
to 25 pages in length) are the most detailed, establishing the issue in a fuller and more contex- 
tualized manner and showing extensive, highly descriptive evidence in favor of action. Some 
of these formats are also known as policy briefs, and the Research in Action component of 
this chapter contains some examples of these forms. 

Researchers can weigh in on a specific piece of proposed legislation or a specific issue or 
topic by providing expert testimony to legislators, either in writing, in person, or both. Ami- 
cus briefs (or amicus curiae, translated as “friend of the court”) are written recommendations 
or arguments submitted to a court in the process of deciding a case. In these briefs, research 
studies can be cited and discussed in the process of making an argument or a recommendation. 

In any and all of these forms, a number of qualities of writing are important. Start and 
Hovland (2004) put forth the AIDA rule to guide such writing: 
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Attract the attention of the intended target audience. 
Spur interest in the evidence being communicated. 
Spark the audience’s desire to act on the issue. 
Propose a specific course of action. 


Research in Action 


In spring 2017, the first author of this book, Erica Scharrer, participated in the Fel- 
lowship Program of the Public Engagement Project at UMass Amherst. The Public 
Engagement Project helps faculty reach broader publics through writing policy 
briefs and networking with advocacy organizations and policy makers; using com- 
munications channels such as blogs, op-eds, or other forms; and practicing being 
a good expert news source in interactions with journalists, www.umass.edu/pep/ 
fellowship-program. 

Here is the one-pager that Prof. Scharrer brought with her for one-on-one discus- 
sions with legislators at the Massachusetts State House, followed by the expert tes- 
timony she submitted in association with a pending state bill. She teamed up with 
representatives of the organization Media Literacy Now for additional visits to the 
state legislators. Check out the advocacy work that Media Literacy Now does to work 
toward establishing media education in K-12 schools through policy formation, https:// 
medialiteracynow.org/. 


Supporting Media Literacy Education In Massachusetts Schools 


Teens spend nearly nine hours per day on recreational media use. Younger children spend 
just under six.' Although there are many positives associated with media use, fake news, 
cyberbullying, commercialism, violence, and stereotypes all pose risks. Media literacy, 
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designed to teach students to access, analyze, evaluate, and create media in all 
its forms’, provides skills for the 21st century. 

I support two bills and would be happy to assist in their creation or implementation. 
Current pending legislation in Massachusetts: 

Bill S278 


* Focuses on civics and news media literacy 

* Calls for Department of Elementary and Secondary Education (DESE) to 
create curriculum piloted on a voluntary basis and assessed 

e Has the goal of requiring news media literacy in all schools 


Bill 3556 


e Spans media literacy and digital citizenship more broadly 

e Calls for DESE to establish an advisory committee and report to Com- 
mittee on Education regarding best practices 

e Asks each city/town/school to develop its own policies and practices for 
implementation and review them annually 


We know media literacy education works. According to research findings, students 
who participated in a media literacy curriculum in MA: 


e Are better able to identify problematic aspects of violent media depic- 
tions (such as violence being accompanied by rewards or being done by 
likeable characters)’ and are more critical of violence in advertising.* 

e Recognize and critique gender stereotypes in movies, TV programs, and 
video.* 

e Understand how advertisers make unhealthy foods and smoking look 
appealing.‘ 

e Write their own “Online Constitutions” for themselves and their class- 
mates to follow to engage safely and civilly on social media platforms.” 


Other states are adopting broad media literacy education standards and practices: 


e Washington has a law that calls for an advisory committee to meet with school 
personnel to develop and implement curricula on safe technology use, digital citi- 
zenship, and media literacy. 

e California has pending legislation to create a model curriculum for grades 1—12 and 
integrate media literacy into social studies curricular frameworks. The Dept. of Ed. 
would make resources and training opportunities available for teachers. 


For more information, contact: 


Erica Scharrer, Professor, Department of Communication, 

University of Massachusetts Amherst. 

‘Common Sense Media 2015; ? Center for Media Literacy; ° Scharrer, Journal of 
Mass Media Ethics; * Sekarasih Olson, Onut, Lanthorn, & Scharrer, Journal of 
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Advertising, > Walsh, Sekarasih, & Scharrer, Journal of Children & Media; ° Bick- 
ham & Slaby, Journal of Children & Media; 7 Scharrer et al, Media education for a 
digital generation. 


Testimony for House Bill 3556 


Senator Chang-Diaz and Representative Peisch, thank you for allowing me to submit 
this written testimony today in support of House Bill 3556, An Act relating to digital 
citizenship and media literacy in public schools. 

My name is Erica Scharrer; I am a professor in the Department of Communication 
at the University of Massachusetts Amherst. I have been a faculty member at UMass for 
17 years, and I study the role of media in the lives of young people. I am the former 
chair of the Children, Adolescents, and Media division of the International Commu- 
nication Association, and I have research expertise on the benefits of media literacy 
education (MLE). 

Graduate students, undergraduate students, and I offer a media literacy program 
at — Elementary School every spring, reaching over 60 sixth graders each 
time. We have also visited a number of additional schools in western Massachusetts 
in the past to carry out and study students’ responses to various media literacy les- 
son plans, including in these locations, xxxxx, XXXXX, XXXXX, XXXXX, XXXXx, and 


XXXXX. 

Based on a review of our research and that of others, we can conclude that media 
literacy education does indeed bring important benefits to students. Some examples of 
evidence from our studies include: 


e After participating in a curriculum designed to promote critical thinking about 
media violence, students became better able to identify problematic aspects of vio- 
lent media depictions (such as violence being accompanied by rewards or being 
done by likeable characters).! 

e A sample of fourth grade and sixth grade students became more critical of violence 
in advertising after participating in MLE. The girls, but not the boys, also became 
more critical of gender stereotypes in ads.’ 

e For two of three conflict scenarios we posed (outlining a hypothetical 
conflict and asking what students would do in that situation), students 
became more likely to choose a non-aggressive response following partici- 
pation in a media literacy program on media violence and interpersonal 
conflict.’ 

e Students demonstrated an ability to recognize and critique stereo- 
types in movies, TV programs, and video games in open-ended writing 
assignments.* 

e After discovering that students were more likely to intervene in cyber- 
bullying if a friend rather than a classmate was implicated, students cre- 
ated “constitutions” to govern their own and others’ online civility and 
safety.’ 


Our own research typically focuses on violence or gender stereotypes, but there 
is also evidence from other researchers that MLE lessons impact students positively 
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on other media-related topics as well, including topics related to news and politics or 
health: 


e David Bickham and Ronald Slaby found fifth grade students became more able 
to understand how advertisers make unhealthy foods and smoking look appealing 
after participating in an MLE curriculum.° They, too, are Massachusetts-based, both 
affiliated with the Center on Media and Child Health at Children’s Hospital Boston. 

e Se-Hoon Jeong and colleagues analyzed 51 studies and found effects of MLE on 
knowledge and attitudes about media, perceptions of media realism and of media 
influence, and ability to critique media. Importantly, positive outcomes were found 
for students of various ages and for various MLE topics. 

e The Jeong and colleagues’ study found that longer MLE curricula were more impact- 
ful, which suggests that if Massachusetts were to develop a systematic, widespread, 
and lasting approach to MLE in schools, positive outcomes would be facilitated. 


From the basis of my expertise, I would make two key recommendations for MLE 
implementation. First, I have found that a curriculum that acknowledges the positive 
aspects of media in addition to pointing out the negatives is important to promote criti- 
cal thinking. Second, I believe that active learning, hands-on application of MLE prin- 
ciples, and interactive dialogue are effective pedagogical approaches for MLE, as well. 

I have been following with interest the example provided in the state of Washington, 
where media literacy education legislation has recently been signed into law. With the 
passing of H3556, Massachusetts would join Washington at the forefront of states pro- 
viding their citizens with the teaching and learning skills that are vital for success and 
well-being in the 21st century. 

In conclusion, I find H3556 to be a comprehensive bill highly responsive to the needs 
of the Commonwealth’s students, educators, and all of its citizens as we face an increas- 
ingly technological and media-driven world. Supporting and developing the ability of 
K-12 students to think critically, deeply, and carefully about the content they create and 
encounter through the internet, social media, television, video games, mobile media, 
audio media, and print is vitally important. I believe we should do everything we can 
to support digital citizenship and media literacy education, and H3556 is an important 
step in that direction. 

Thank you for the opportunity to testify, and I am happy to assist in any way that you 
and your colleagues would find useful. 


Contact information: 


Erica Scharrer, PhD; Department of Communication; $330 Integrative Learning 
Center University of Massachusetts Amherst; Amherst, MA 01003 

"Scharrer, Journal of Mass Media Ethics;? Sekarasih, Olson, Onut, Lanthorn, & 
Scharrer, Journal of Advertising;? Scharrer, Journal of Media Literacy Education;‘ 
Walsh, Sekarasih, & Scharrer, Journal of Children & Media;> Scharrer et al., 
Media education for a digital generation; Bickham & Slaby, Journal of Children & 
Media;’ Jeong, Cho, & Hwang, Journal of Communication;’ see www.usnews. 
com/news/best-states/ washington/articles/2017-04-21/new-law-promotes- 
media-literacy-internet-safety-in-schools. 
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Examples of Quantitative Communication Research Connected to Policy 


Communication researchers and those from allied disciplines have provided evidence- 
based perspectives on a number of issues and topics at stake in public policy formation. 
Here is a handful of examples of the ways that quantitative research, in particular, has been 
used to address communications and media policy questions. We recognize that this list is 
U.S.-based and therefore limited in scope, but we hope it will illustrate this type of work, 
nonetheless. 


e Northwestern University Professor Ellen Wartella was a member of the U.S. Surgeon 
General’s Scientific Advisory Committee on the Causes of Youth Violence in 2000 and 
has testified before Congress on multiple occasions, including regarding the Children and 
Media Research Act in 2004, Children’s Television Practices Act of 1988, and the Chil- 
dren’s Education TV Act of 1985, and she testified before the Federal Trade Commission 
on Children and Television Advertising in 1979. 

e University of Arizona Professor Dale Kunkel has served as a Congressional Science Fel- 
low and has also testified on a number of media and children-related topics and initiatives, 
including before the U.S. Senate Committee on Commerce, Science, and Transportation 
on the effects of media violence on children in 2007. 

e Media psychologists submitted amicus briefs on both sides of the issue when the USS. 
Supreme Court ruled in 2011 on a California law that banned the sale of M-rated (M for 
Mature) video games to minors. 

e Rutgers University Department of Journalism and Media Studies Professor Amy Jordan 
has studied outcomes associated with the Telecommunications Act of 1996 mandate that 
television sets contain the v-chip, a computer chip able to block out programs deemed 
too sensitive for children by parents and caregivers, such as those containing violence. 
This research was also cited in an amici curiae signed by the American Academy of Pediat- 
rics, the Parent Teacher Association, and other organizations in 2008 and presented to the 
Supreme Court to provide context about children and media policy as the Court decided 
the Federal Communications Commission v. Fox Television Stations case. 

e Catherine J.K. Sandoval (2011) of the Santa Clara University School of Law studied 
FCC-supplied data for patterns associated with media ownership by racial minorities in 
collaboration with the advocacy organization Free Press. An important observation from 
this work found that when television and radio stations are owned by people of color, the 
content itself tends to be more diverse. 

e Graciela León Orozco (2011), Associate Professor and Coordinator of the School 
Counseling Program in the Department of Counseling at San Francisco State Uni- 
versity, conducted a mixed methods (survey and interview) analysis of the ways in 
which one particular radio station was serving the needs of the local community in 
Appalachia. 

e The research on Muslim Americans’ coping responses to media stereotypes conducted by 
Muniba Saleem of the University of Michigan and Srividya Ramasubramanian of Texas 
A&M and the second author of this book published in 2019 was cited as part of two ami- 
cus curiae briefs submitted to the Ninth Circuit court and the U.S. Supreme Court in the 
State of Hawaii v. Trump case 585 challenging the travel ban against Muslim-dominated 
countries. 

e The Spotlight Scholar feature of this chapter includes Professor Karen Dill-Shackleford’s 
reflections on her experience providing expert testimony on misogyny in rap music 
lyrics. 
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Despite the historical presence (and the continuing promise) of such efforts, in the book Com- 
munications Research in Action: Scholar-Activist Collaborations for a Democratic Public Sphere, Philip 
M. Napoli and Minna Aslama (2011) point out that: 


Communication researchers’ interactions with—and contributions to the work of—the 
public interest and advocacy communities has historically been sporadic at best, with 
many analysts identifying a persistent scholar-activist divide that has limited the extent to 
which communications researchers and advocates work collaboratively to solve pressing 
social problems. 


(p. 1) 


Indeed, the benefits as well as the drawbacks of researcher-advocacy organization partner- 
ships are well illustrated in the observations of Katherine Heinz, Professor of Communica- 
tion at Santa Clara University, and Christina Romano Glaubke, of the advocacy organization 
Children Now (2011). They teamed up to submit evidence-based comments to the FCC 
on the ways that easing media ownership rules affects TV program offerings for children. 
They report, “This voice did not go unheeded. When the FCC issued their rulemaking on 
June 2, 2003, they adopted one of Children Now’s recommendations and inserted the only 
public interest provision to protect educational programming for children” (p. 75). Heinz 
and Romano Glaubke worked together once again to gather content analysis data compar- 
ing children’s programs in seven local markets from 1998 to 2006. Heinz discusses both the 
frustrations and the triumphs that the partnership provided in helping research have an impact 
in conversations with industry executives: 


This is not to say that discussions were always fruitful, or even cordial, but they happened. 
And I’m quite sure that they wouldn’t have—or I wouldn't have been a part of them—if 
it weren't for the partnership developed between me and Children Now. I have no illu- 
sions that I would have sat at the table with heads of ABC Entertainment, Nickelodeon, 
or the WB Network to discuss the media in the lives of children if it had not been for this 
collaboration. After all, industry folks don’t pick up the Journal of Communication. 

(Heinz & Romano Glaubke, 2011, p. 82) 


Summary 


Publication of research in the form of a book, chapter, or journal article typically brings 
with it a seal of approval that the research in question is solid, trustworthy, and credible. In 
this chapter, we have walked through the main components of a quantitative research project 
earmarked for publication in an academic journal and discussed, as well, both the merits and 
the drawbacks of the process of peer review. Titles, introductions, literature reviews, methods 
sections, results, tables and figures, discussion sections, and references are the key components 
of a quantitative paper. They are used to present an original research study and place it within 
the larger context of the existing knowledge on the topic. In doing so, knowledge builds piece 
by piece, study by study. 

Yet, “even the most brilliant scientific paper becomes something useful only when it is 
planted in the mind of someone with the power to do something about it” (Bennett & Jessani, 
2011, p. 129). We have discussed in this chapter, as well, the ways in which Communication 
researchers can collaborate with others and/or work independently to bridge the “know-do” 
gap, bringing evidence and data in conversation with policies that have been formed or are 
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currently forming. Such efforts provide excellent opportunities for research to have an impact 
at the same time that they demand humility, flexibility, and the investment of time and effort 
on behalf of researchers. They also demand that the researcher be versatile in the manner in 
which they communicate, depending on the audience and the context in which they are 
communicating. 

Finally, we have discussed avenues to bring scholarly insights and evidence to the attention 
of the public through legacy and digital media outlets. For traditional news media in addition 
to blogs, websites, and social media posts, messages should be crafted with the interests and 
needs of audiences in mind as well as based on the logic of particular communication chan- 
nels. Relationships with journalists and engagement with the public are built on connection, 
listening as well as speaking, and establishing credibility, reliability, and trust. Meaningful 
action is co-created. 

Overall, then, there are opportunities and challenges for researchers attempting to make 
research useful for reaching change makers, members of advocacy groups, policy makers, 
and/or the wider public. As Badgett (2015) writes, even when academics are committed 
to public engagement, “their heavy workloads, the lack of support for engagement, and 
the decline of tenure-track jobs . .. make public engagement difficult” (p. 16). We hope 
this chapter has provided tools, considerations, information, and perhaps even inspiration 
useful to researchers interested in navigating writing and communicating for publication, 
policy, and the public. Indeed, for social justice researchers, these efforts are particularly 
important since they ensure a close connection between the “scholarly” world and the 
world around us. 


Discussion Questions and Activities 


1. Finda research article that was published in an academic journal on a social justice-related 
topic of interest to you. Think about how the article is organized through the use of 
headings and subheadings. Read each section and see whether the authors have included 
the information you would expect to find, given the discussion in this chapter. 

2. Find two or three academic journals that you think might publish research in your area 
of interest. Read about the journals on their websites, studying the editorial board, aims 
and scope, and other features. Then, look up a recent article published in each journal 
or, better yet, an entire issue of the journal to see what sorts of research the journal is 
publishing. 

3. What do you think are the pros and cons (or opportunities and challenges) of publishing 
research in traditional academic outlets like scholarly journals compared to disseminat- 
ing research online through blogs or social media? If you plan to be a researcher for your 
career, how will you approach these options? 

4. Find a blog on a social issue that you care about written by an academic, and study it for 
the manner in which research is discussed. What tone and style does the author use? How 
does the author discuss evidence? What visuals or other features are there? Did the blog 
spur a dialogue, as seen through comments? 

5. Find a partner and role play an interview between a journalist and a researcher who is 
serving as an expert news source, switching roles after the first role play so that each of 
you has a turn to be the journalist as well as the researcher. When you are the journal- 
ist, anticipate the sort of questions that might be of interest to a news audience. When 
you are the researcher, think about being clear, concise, and engaging, redirecting when 
necessary and otherwise following the advice offered in the chapter. 
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